atelet: MemoryPullCache retains full image tarballs indefinitely (LRU bounded by entry count; the 100 MiB guard measures manifest size) #437

Summary

Two bugs in cmd/atelet/internal/memorypullcache. Together they let atelet's heap grow without bound — it permanently retains one fully-extracted copy of every distinct image digest it has ever pulled — until the kernel OOM-kills it.

This is distinct from #120, which is a transient spike inside a single mutate.Extract. What follows is permanent retention across pulls.

Bug 1: the size guard measures the wrong thing

size, err := img.Size()
if size > 100*1024*1024 {
    slog.InfoContext(ctx, "Image is too large to cache", ...)
    return mutate.Extract(img), &imageCfg, err   // not cached
}

v1.Image.Size() returns the size of the manifest (a few KB), not the size of the image content. The guard therefore never fires: Image is too large to cache appears 0 times in our logs, while a ~740 MiB (extracted) image is cached on every pull.

Bug 2: the LRU is bounded by entry count, not by bytes

cache: lru.New(256)

type cachedImage struct {
    tar []byte
    cfg v1.Config
}

Each entry holds the entire flattened, uncompressed image tarball. Bounded at 256 entries, atelet will retain 256 × image-size before evicting anything. The existing TODO already anticipates this:

// TODO: Need a smarter cache with bounds on total consumed size, not just number of entries.

Impact (measured)

A Node-based coding-agent image: 222 MiB compressed (4 layers) → ~740 MiB extracted.

• go_memstats_heap_inuse_bytes = 1.455 GiB with exactly 2 distinct image digests cached (2 × ~0.74 GiB). go_goroutines = 42 and container_memory_cache stays flat at 0.02–0.08 GiB, so this is live, reachable heap — not garbage, not page cache.
• Over ~4 days atelet RSS climbed monotonically from 0.8 GiB to 15.3 GiB and was OOMKilled (exit 137). The sawtooth then repeats.

Growth is per distinct digest, not per actor — cache hits cost nothing. But since each actor pins the image digest it was created with, digest diversity accumulates as older actors are resumed, so any cluster that ships new image builds keeps introducing new permanently-retained entries.

Collateral: actors stranded in STATUS_RESUMING

When atelet dies mid-restore, the in-flight RestoreWorkload is cancelled — while creating pause container: while running 'runsc create': context canceled — and the actor is left in STATUS_RESUMING indefinitely. An explicit resume recovers it, but it does not self-heal.

Suggested fix

Make the cache size-aware: account len(tar) and bound total bytes (configurable budget), evicting LRU. Hot digests stay cached, so resume latency does not regress, while the heap is capped.

Fixing only Bug 1 would also stop the growth, but it would disable caching for exactly the images that matter and add a registry pull to every resume — which makes #228 worse. Bounding by bytes seems strictly better.

Happy to send a PR if this direction sounds right.