Repository navigation
ateom sandbox runtime output (runsc, pause container, cloud-hypervisor) has no actor attribution #2271
Description
What happened?
On multi-actor workers, sandbox runtime subprocess output (runsc and the pause container in ateom-gvisor, cloud-hypervisor in ateom-microvm) goes to the shared worker pod log without ate.actor.* attributes. In ateom-microvm it also carries a stale trace_id from the boot RPC. Runtime warnings and errors cannot be attributed to a specific actor.
1. ateom-gvisor writes runsc and pause container output to raw os.Stdout / os.Stderr
runschelper commands (cmd/ateom-gvisor/runsc.go):cmdCheckpoint,cmdFsCheckpoint,cmdPause,cmdResume,cmdDelete,cmdState,cmdList,cmdKill, andcmdWaitsetcmd.Stdout = os.Stdoutandcmd.Stderr = os.Stderr.- Pause container (
cmd/ateom-gvisor/main.go):RunWorkloadandRestoreWorkloadpassos.StdouttocmdCreate,cmdStart, andcmdRestoreforocispec.PauseContainer. Those helpers assign that writer to bothcmd.Stdoutandcmd.Stderr.
2. ateom-microvm slogWriter has no ate.actor.* attributes and keeps the boot trace_id
coldBootActor (cmd/ateom-microvm/run.go) and restoreFullScope (cmd/ateom-microvm/restore.go) call ch.LaunchVMM with slogWriter{ctx} as the Stdout and Stderr of cloud-hypervisor. slogWriter (cmd/ateom-microvm/run.go) stores the context it gets at VM start and writes each output chunk with slog.InfoContext under the message cloud-hypervisor, with no other attributes.
- No
ate.actor.*attributes:slogWriterlogs noate.actor.uid,ate.actor.name, orate.actor.atespace, so the output of concurrentcloud-hypervisorprocesses on one worker cannot be told apart. - Stale
trace_id/span_id: The stored context is theRunWorkloadorRestoreWorkloadRPC context. Each latercloud-hypervisorlog line, for example duringCheckpointWorkloadorTerminateWorkload, carries thetrace_idandspan_idof that ended span.
Activity
gambtho commentedon Oct 8, 2026
Ran into this while testing suspend/resume on a small cluster. We couldn't tell from the logs which actor a micro-VM's output belonged to, so we couldn't say whether a resume was a fresh start or a restore.
The actor-tagged starting/restoring logs on main mostly cover that now. The cloud-hypervisor output still goes through
slogWriter{ctx}with no actor on it though, and it keeps the startup span after startup is done.I'm happy to send a small PR for the micro-VM side (boot and restore paths, plus a test with two actors logging at once) if that works for you. gVisor could come after.
cloudonly commentedon Oct 8, 2026
The gVisor side works for me — I'll take it, thanks for leaving it.
Plan, matching the envelope approach the app containers already use (
StartJSONLogPipe+WrapContainerLogs):RunWorkload/RestoreWorkloadcurrently passos.StdouttocmdCreate/cmdStart/cmdRestore; switch them to a per-actor pipe labeledate.actor.container.name="_pause". Small and mechanical.cmdCheckpoint..cmdWait): same envelope via a per-invocation pipe, with attribution carried on the runsc struct socontainerRuntimeand its fakes don't change.cmdListkeeps its parsed stdout buffer; only stderr is re-routed. Sequencing note: Move static runsc flags to configuration file #2281 rewrites the flag blocks in those same helpers — happy to rebase on top of it or wait, whichever is easier for reviewers.One thing worth knowing for the micro-VM side: besides missing actor attributes,
slogWriterholds the boot RPC's context, so cloud-hypervisor output during later checkpoint/terminate carries the ended boot span's trace_id/span_id. A replacement writer that holds no context drops that by construction — that's also how the app-container forwarding works (oncontext.Background()).No registry changes needed on either side: the labels are code-defined (
ate.actor.container.name), and the cardinality rule bars actor identity from metric labels only.cloudonly commentedon Oct 8, 2026
gVisor part 1 is up: #2330 — the pause container (sentry/gofer) now gets its own actor-log pipe under
ate.actor.container.name="_pause", same envelope as the app containers. The one-shot runsc helpers follow separately once #2281 lands or if reviewers prefer them on top of it.ateom-gvisor: attribute runsc output for the pause container (#2330)