[Security] imagecache serves any cached image by digest alone: a template can run another tenant's image under any repository name with no pull authorization #3017

Summary

internal/imagecache keys cached images by manifest digest alone. When an ActorTemplate names an image by digest, EnsureImage parses the digest out of the reference and returns the cached image if any image with that digest is already in the node's pool, without ever contacting the registry named in the reference (internal/imagecache/imagecache.go:433-473, and the README: "digest refs hit the cache with no network I/O"). The repository and registry in the reference are therefore not checked at all for cached digests.

Consequences on main 4b4fbb2, reproduced on a 3-node GKE cluster:

1. Any tenant who knows the digest of an image that any other tenant (or the system) has run on a node can run that image, from any atespace, under any made-up repository name, with no registry credentials and no pull authorization. This includes images that were pulled with the node's GCP credentials (Support per-atespace image pull credentials and cache authorization in atelet #2080) or with a per-atespace credential provider: once cached, the image is a node-wide unauthenticated image store.
2. An ActorTemplate can claim to run gcr.io/my-org/trusted@sha256:X while actually running whatever X is, because nothing verifies that gcr.io/my-org/trusted ever contained X. Audit logs and templates misstate provenance.

Reproduction

Node pool already had busybox@sha256:1487d0af... (from another atespace's template) and the private gcr.io/gke-ai-eco-dev/ate-images/glutton-...@sha256:f40229be... (pulled with node credentials) in the cache.

# template 1: repository does not exist, digest is busybox's
containers:
- name: s
  image: gcr.io/gke-ai-eco-dev/ate-images/nonexistent@sha256:1487d0af5f52b4ba31c7e465126ee2123fe3f2305d638e7827681e7cf6c83d5e
# template 2: a different registry and name, same digest
  image: docker.io/library/nginx@sha256:1487d0af5f52b4ba31c7e465126ee2123fe3f2305d638e7827681e7cf6c83d5e
# template 3: made-up name, digest of the private glutton image
  image: example.com/attacker/whatever@sha256:f40229be1a46e69bbf3abda041c8e1166395411c8061e5820370aef19d15263c
# control: made-up name, digest not in the cache
  image: gcr.io/gke-ai-eco-dev/ate-images/nonexistent@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef

kubectl ate create actor-template -f ..., create actor, resume actor for each:

atelet logs (jsonPayload.ref is the reference from the template, digest the cache key):

22:10:19.590  atelet-98z9z  Image cache hit   ref=gcr.io/gke-ai-eco-dev/ate-images/nonexistent@sha256:1487d0af...  digest=sha256:1487d0af...
22:16:33.153  atelet-98z9z  Image cache miss  ref=gcr.io/gke-ai-eco-dev/ate-images/nonexistent@sha256:01234567...  digest=sha256:01234567...   (control, then pull fails)
22:16:41.292  atelet-98z9z  Image cache hit   ref=docker.io/library/nginx@sha256:1487d0af...                         digest=sha256:1487d0af...
22:16:45.911  atelet-v97bm  Image cache hit   ref=example.com/attacker/whatever@sha256:f40229be...                   digest=sha256:f40229be...

The same happens for golden-snapshot builds: the golden actor for template 1 started from the cached busybox under the nonexistent name (it then crashed for an unrelated reason).

Where

• internal/imagecache/imagecache.go:445-447: a name.Digest ref is turned straight into the cache key; cachedImageHit(digest) at :466 returns the pooled image. The withCredentials/remote.Head path at :452-463 runs only for tag refs.
• The pool is node-wide and shared across atespaces by design (internal/imagecache/README.md); [Security] Image Layer Cache Poisoning via Unverified DiffID Keying #2082 fixed layer content verification, but the record lookup is still by digest only.

Expected

For a digest ref the cache should only be used after the caller's access to the named repository has been established, for example a remote.Head (or a manifest GET) against repo@digest with the credentials that apply to this atespace, which also proves the repository contains that digest. The HEAD is one small request per resume and is already the cost paid for tag refs; it could be cached per (atespace credential, repo, digest) for a bounded time if latency matters. Alternatively scope cache records by (repository, digest) and only share the underlying layer pool, which is content-addressed and already verified.

Related

#2080 (node credentials used for all tenant pulls) makes the cache easier to populate; #2082 (closed) covered poisoning of the layer pool, not the lookup key.