Repository navigation
Support per-atespace image pull credentials and cache authorization in atelet #2080
Description
Severity: MEDIUM (2.56) | CWE: CWE-863 (Confused Deputy)
Current Status
- Solved on
main: PR #917 (e54d51c7) removed unconditional ambient GCP metadata keychain usage frominternal/imagecache/imagecache.go.ateletnow defaults--enable-ambient-registry-auth=false(cmd/atelet/main.go:88) and usesauthn.DefaultKeychainunless explicitly enabled. - Remaining open work:
- When
--enable-ambient-registry-auth=trueis enabled (currently used byate-setupon GKE so nodes can pull private images from the project's Artifact Registry),ateletuses the node's ambient GCP service account for anygcr.ioor*.pkg.devimage pull across allatespaces. ateletdoes not yet support per-atespaceor per-actor image pull credentials / Workload Identity bindings, nor does it authorize local image cache hits by digest across tenants (see also #3017).
- When
Problem Description
In remoteOpts (internal/imagecache/imagecache.go:768-794), imagecache attaches the host node's ambient Google Cloud credentials when pulling images from Google Container Registry (gcr.io) or Artifact Registry (pkg.dev):
- When a tenant submits an
ActorTemplateor workload specifying an image hosted ongcr.ioorpkg.dev, the pull operation unconditionally injects the node's Google Service Account (GSA) auth token. - No check verifies whether the tenant requesting the image has permission to access that specific repository or project.
Attack Vector
A tenant user can pull private images belonging to the cluster operator or other projects:
- Tenant defines an
ActorTemplatepointing to a private internal image (e.g.,us-docker.pkg.dev/internal-project/private-repo/secret-image:latest). - Node daemon uses its node-level IAM credentials to pull the image layers onto the worker host.
- The tenant's actor launches using the private image, exposing proprietary code, internal tooling, or embedded secrets.
Impact
Confused deputy privilege escalation allowing low-privileged tenants to access private container images readable by the underlying GKE/GCP node service account.
Proposed Remediation
- Do not use node ambient credentials for tenant-specified image pulls; require explicit tenant image pull secrets or Workload Identity bindings.
- Restrict default registry access or maintain an allowlist of public image repositories.
- Validate tenant authorization to target images before initiating pull requests.
Activity
EItanya commentedon Oct 1, 2026
I believe this should be fixed by #917 Benjamin Elder (@BenTheElder)
BenTheElder commentedon Oct 1, 2026
I don't think so. We moved it from the node identity to the atelet identity basically, but we still have effectively node-level auth instead of per tenant, e.g. from the problem description:
That's still true. And I don't know that we have a reasonable plan to sort it out before GA.
zlammerts-svg commentedon Oct 2, 2026
Benjamin Elder (@BenTheElder) Will you be able to take this one as well given your recent changes related to
ateletimage pull credentials?BenTheElder commentedon Oct 2, 2026
I think we still need a determination if this is in scope and what exactly we want to do. It might be pretty involved to design something actor aware and overlap with the ongonig substrate authz work.
EItanya commentedon Oct 3, 2026
Agreed, this overlaps with the
ActorTemplateimage_pull_secretsdesign that I just pulled out of M3. I think we can come up with a solution here but it will come down to prioritization.ahmedtd commentedon Oct 3, 2026
I agree that we are likely not going to be able to fix this before GA.
zlammerts-svg commentedon Oct 6, 2026
Thanks all, removing this from M3 now.
BenTheElder commentedon Oct 17, 2026
We need to decide if we'll expand the atelet scoped credential options before resolving actor scoped credentials. I think it depends on timing. We do need to give people a solution but static secrets seem less mitigable than credential plugins (we could start passing actor info to them?)
aditya-shantanu commentedon Oct 22, 2026
Folding #3017 into this issue as a sub-case, per Ben's question there. The cache makes the missing per-tenant image authorization cheaper to exploit than a pull:
internal/imagecachekeys cached images by manifest digest alone (EnsureImage,imagecache.go:445-466), so once any tenant or the system has pulled an image onto a node, an ActorTemplate naming<any registry>/<any repo>@<that digest>runs it with no registry call at all. Reproduced on main 4b4fbb2 three ways (nonexistent repo, different registry name, and the private glutton image underexample.com/attacker/whatever); e2e reproduction on a private branch: https://github.com/aditya-shantanu/substrate/blob/e2e/coverage-gaps/internal/e2e/suites/demo/imagedigest_test.go. Whatever shape the per-tenant authorization takes (image pull secrets, actor-aware credential plugins), it has to gate digest cache hits too, not only pulls: a HEAD againstrepo@digestwith the tenant's credentials before serving from the pool would do it.[-][Security] Node GCP Credentials Used Unconditionally for Tenant Image Pulls[/-][+]Support per-atespace image pull credentials and cache authorization in atelet[/+]BenTheElder commentedon Nov 3, 2026
Yes, that's tablestakes and a notorious vector vector from Kubernetes Greg will also be familiar with. There's a relatively recent KEP to fix this without requiring imagePullPolicy always, finally.