Large image pulls death-loop in ResumeActor and amplify atelet RSS #230

Expected Behavior

When ate-controller reconciles a new ActorTemplate, the golden-actor ResumeActor RPC should complete and the template should transition through PhaseResumeGoldenActor → PhaseWaitGoldenActor → PhaseReady. For any image whose cold pull + extract + gVisor restore can reasonably complete, this should happen within one or a few reconcile attempts.

atelet RSS during this path should be bounded by the image's working-set size, regardless of whether individual ResumeActor calls succeed or are retried.

Actual Behavior

For container images whose cold-pull path (registry fetch + decompress + extract + gVisor restore) exceeds ~28 seconds:

1. Death loop on ResumeActor. Every call returns DeadlineExceeded after ~28s and ate-controller's reconciler retries with exponential backoff (~30-60s). The ActorTemplate sticks in PhaseResumeGoldenActor and never reaches PhaseReady. The deadline is hardcoded at cmd/ateapi/internal/controlapi/workflow.go:145 via acquireActorLock(ctx, id, 30*time.Second, 2*time.Second), which couples the Redis lock TTL and the workflow deadline into a single 28s budget.

2. Atelet RSS amplification. Each cancelled pull leaves layer-blob HTTP goroutines on atelet blocked in Body.Read holding partial-blob buffers, because internal/memorypullcache/memorypullcache.go:123-136 doesn't pass the caller's ctx into remote.Image. RSS amplifies proportionally to the loop duration. Observed climb-then-recede shape on a kind node:

   atelet   136m   2453Mi    # baseline
   ...
   atelet   195m   9410Mi    # peak during loop — partial-blob buffers from cancelled pulls
   atelet   588m   6791Mi    # receding as orphaned HTTP goroutines finally drain
   

   Not a permanent leak: orphaned goroutines drain when their TCP connections eventually drop or when a retry's pull happens to complete. But on a production node that's already tight, the spike is enough to OOM-kill atelet — taking every other actor on that worker down with it, not just the one being resumed.

Observed ate-controller output during the loop:

ERROR Reconciler error ... error: "while resuming golden actor: rpc error:
code = DeadlineExceeded desc = context deadline exceeded"

Steps to Reproduce the Problem

1. Set up kind + install substrate:

   ./hack/create-kind-cluster.sh
   ./hack/install-ate-kind.sh --deploy-ate-system

2. Build the ateom-gvisor image into the kind-local registry and capture the resolved reference:

   export KO_DOCKER_REPO=localhost:5001
   export KO_DEFAULTPLATFORMS=linux/$(go env GOARCH)
   ATEOM_IMAGE=$(./hack/run-tool.sh ko build -B ./cmd/ateom-gvisor)
   echo "ateom image: $ATEOM_IMAGE"   # localhost:5001/ateom-gvisor@sha256:...

3. Install metrics-server (needed for kubectl top to verify atelet RSS):

   kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/high-availability-1.21+.yaml
   kubectl scale -n kube-system deploy/metrics-server --replicas=1
   kubectl patch -n kube-system deploy metrics-server --type=json \
     -p='[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]'
   kubectl rollout status -n kube-system deploy/metrics-server

4. Apply the ActorTemplate with the offending image:

   kubectl apply -f - <<EOF
   apiVersion: v1
   kind: Namespace
   metadata:
     name: ate-repro
   ---
   apiVersion: ate.dev/v1alpha1
   kind: WorkerPool
   metadata:
     name: repro-pool
     namespace: ate-repro
   spec:
     replicas: 1
     ateomImage: ${ATEOM_IMAGE}
   ---
   apiVersion: ate.dev/v1alpha1
   kind: ActorTemplate
   metadata:
     name: repro-fat-image
     namespace: ate-repro
   spec:
     workerPoolRef:
       name: repro-pool
       namespace: ate-repro
     runsc:
       amd64:
         url: "gs://gvisor/releases/nightly/2026-05-19/x86_64/runsc"
         sha256Hash: "a397be1abc2420d26bce6c70e6e2ff96c73aaaab929756c56f5e2089ea842b63"
       arm64:
         url: "gs://gvisor/releases/nightly/2026-05-19/aarch64/runsc"
         sha256Hash: "1ba2366ae2efceba166046f51a4104f9261c9cb72c6db8f5b3fe2dc57dea86b9"
     pauseImage: "registry.k8s.io/pause:3.10.2@sha256:f548e0e8e3dc1896ca956272154dde3314e8cc4fde0a57577ee9fa1c63f5baf4"
     containers:
       - name: fat
         image: ghcr.io/kagent-dev/nemoclaw/sandbox-base@sha256:d52bee415dc4c0dba7164f9eabe727574c056d4f211781f20af249707883a3b4
         command: ["/bin/sh", "-c", "sleep 3600"]
     snapshotsConfig:
       location: gs://ate-snapshots/repro/
   EOF

   No manual kubectl ate resume actor needed. ate-controller calls ResumeActor against the golden-actor ID as part of the AT reconcile — that's enough to exercise the bug.

5. Confirm the symptoms:

   # Death loop in ate-controller
   kubectl logs -n ate-system deploy/ate-controller --tail=0 -f | grep -i "deadline\|golden"
   
   # ActorTemplate stuck (expect: ResumeGoldenActor)
   kubectl get actortemplate -n ate-repro repro-fat-image -o jsonpath='{.status.phase}{"\n"}'
   
   # atelet RSS climb-then-recede
   while true; do date; kubectl top pod -n ate-system -l app=atelet --containers; sleep 5; done

If the bug does not trigger on your bandwidth: the cold-pull path completes inside 28s on fast corp LAN / warmed registry mirrors. Force it by throttling the kind node (docker exec kind-control-plane tc qdisc add dev eth0 root tbf rate 10mbit burst 32kbit latency 400ms — apt-get install -y iproute2 first if tc is missing) or by swapping image: for something much larger (e.g. nvcr.io/nvidia/pytorch:24.05-py3 at ~12GB). The death loop sometimes escapes via bandwidth variance + HTTP keep-alive warmup; watch for at least 5 minutes before concluding it didn't reproduce.

Specifications

• Version: HEAD of main
• Platform: kind (single-node, control-plane only), reproduced on darwin/arm64; expected to reproduce on any platform where the cold-pull path for the chosen image exceeds ~28s