atelet drops all capabilities, so it cannot remove a DurableDir populated by a non-root workload #2034

Summary

atelet removes an actor's DurableDir volume on delete with an in-process os.RemoveAll. The atelet container runs as uid 0 with capabilities: {drop: [ALL]}, so ordinary permission checks apply to it. Once a non-root workload has written files into its DurableDir, the removal fails with permission denied, and the actor sticks in DELETING while holding its worker. Adding DAC_OVERRIDE and FOWNER to atelet fixed this on a test cluster.

Impact

• Who hits it: every integrator running a non-root workload that writes to its DurableDir. That is the normal pattern for an agent workspace.
• What breaks: on stock d277088b, every such delete fails, and each one leaks a worker. It was observed on 4/4 actors with populated workspaces.
• Cost: a DaemonSet patch outside the supported install, or manual force-clean for every delete.
• Severity: High.
• Proposed priority: P0. It happens deterministically on a common workload pattern and consumes pool capacity with every delete. The fix is small and was verified on a test cluster.

Environment

• Substrate d277088b (v1alpha).
• GKE 1.36.x.
• gVisor sandbox class, unprivileged WorkerPool workers.
• Stock atelet DaemonSet from the install. The stock DaemonSet still carried drop: [ALL] on two later test runs.

Steps to reproduce / evidence

1. Create an actor with a DurableDir volume (verified live) mounted at /workspace, running as uid 0 by default. The workload drops to uid 1000, chowns its workspace to itself and writes files into it.
2. Where the mount point comes from (code-read at d277088b). atelet creates the mount point as 0700 in prepareOCIBundles, at cmd/atelet/main.go:1433: os.MkdirAll(volPath, 0o700). After the workload's chown, the host-side tree is 700 1000:1000.
3. Delete the actor (verified live). atelet fails:

   Internal: failed to remove actor directories during terminate ... TERMINAL_FILE_SYSTEM_ERROR: while deleting durable-dir volumes mount dir: %w: openfdat /var/lib/ateom-gvisor/actors/<uid>/durable-dir/workspace: permission denied
   

   The ... marks an elided prefix; the rest is verbatim, including the literal %w: (see Additional detail). runsc delete itself succeeds (in ateom). The actor stays in DELETING and keeps its worker.
4. Why it fails (code-read at d277088b plus live /proc):
   • The removal is os.RemoveAll(durableDirVolumesMountDir) at cmd/atelet/main.go:1912-1913 (path assignment at :1912, the call at :1913). There is no helper process, no setfsuid and no user-namespace switch.
   • Live /proc/<atelet-pid>/status shows Uid: 0 0 0 0 and CapPrm = CapEff = CapBnd = 0, in the host user namespace. uid_map is 0 0 4294967295.
5. Discriminating runs (verified live):
   • Workspace root chmod'ed to 755: the delete still fails. All 10,860 failures are unlinkat EACCES on the entries inside; the openat now succeeds.
   • Workspace emptied and left at 700 1000:1000: the delete succeeds in 0.35s. rmdir only needs write and search permission on the 755 0:0 parent.
6. Fix verified (verified live). Patch the atelet DaemonSet container with exactly:

   securityContext:
     capabilities:
       drop: [ALL]
       add: [DAC_OVERRIDE, FOWNER]

   Live CapEff becomes 0xa. A populated 700 1000:1000 workspace then deletes in 0.80s with no EACCES, and the worker is freed. On two later test runs the same patch was applied again, and every actor delete tested succeeded. The patch was reverted after each run.

Expected vs actual

• Expected: deleting an actor removes its DurableDir whatever uid the workload used, and the delete completes.
• Actual: removal fails with EACCES once a non-root workload has written files. The actor sticks in DELETING, holding its worker.

Workaround

An integrator has two options:

• Patch the atelet DaemonSet to add DAC_OVERRIDE and FOWNER (step 6). This must be reapplied after reinstalling or upgrading Substrate.
• Force-clean each actor: delete the worker pod, wait for CRASHED, then call DeleteActor again. This leaves the host directories behind (see Force-clean … leaks host actor directories).

Additional detail

Suggested fixes (any one works):

• Add CAP_DAC_OVERRIDE and CAP_FOWNER to the atelet container. That is the minimum for RemoveAll of trees with arbitrary owners.
• Or perform the removal in a context that already has them, for example in ateom, which runs runsc delete with DAC_OVERRIDE.
• Or chown the tree back to atelet's uid before removing it.

Minor, cosmetic: the literal %w: in the error comes from cmd/atelet/main.go:1914, which passes a message containing %w to wrapFileSystemErr. That helper (cmd/atelet/sandbox_assets.go:499-504) formats the message with %s, so the verb is printed as text.

The error is returned to the DeleteActor caller, but the actor is left in DELETING with its worker held, because the worker is released only when atelet terminate succeeds (cmd/ateapi/internal/controlapi/workflow_delete.go:75-86); see A failed actor delete leaves the actor in DELETING indefinitely….

Related issues:

• A failed actor delete leaves the actor in DELETING indefinitely, holding its worker (failed-delete-stuck-in-deleting-holds-worker.md).
• Force-cleaning a stuck actor (worker pod delete, then DeleteActor) leaks host actor directories (force-clean-leaks-host-actor-dirs.md).
• Image-layer file ownership appears as uid 0 inside the actor (image-layer-file-ownership-lost.md): the missing capabilities in atelet also bear on image unpacking.