Actor resource telemetry Phase 1: atelet reader and OTel emission #896

What

Phase 1 of #550: atelet polls the GetWorkloadStats RPC that Phase 0 (#594) put on both ateom runtimes, and turns the samples into telemetry — template-level metrics for the TSDB, actor-level events for the log stream. After this phase, "how much CPU/memory is this actor using" has an answer you can query, not just an RPC you can call.

Phase 0 delivered the server side: #667 (RPC + attribution retention), #739 (gVisor cgroup read), #832 (micro-VM guest-agent read, in review). Nothing calls it yet.

Emission model

Two channels, split exactly along #174's cardinality decisions:

• Metrics (TSDB): aggregated per ActorTemplate on each node before export. Labels: ate.template.namespace, ate.template.name, ate.sandbox.class, stats source — all bounded sets. Actor and atespace identity never appear as metric labels. (internal/ateattr's metric-label comment should be reconciled with Observability At-Scale #174 decision 2 while here: template keys are sanctioned labels but currently documented in the identity block.)
• Events (logs): per-actor samples as structured log events carrying the full identity echoed by the RPC (ate.actor.uid, ate.actor.name, ate.atespace, template, class, source, the four measurements, observed_at) — same stream and label vocabulary as actorlog's lifecycle events. Queryable per actor and per atespace in the log store; never a time series.

Transport, to be explicit: metrics ride the existing OTLP push (serverboot.InitMetricsPushOnly → the collector in ate-otel-config.yaml) — new instruments on a pipeline atelet already runs. Events do NOT go to the collector in this phase: like every event in this system they are structured JSON on stdout, picked up by the node logging agent, landing in the same store as the lifecycle events they join with. Nothing here uses the OTel Logs SDK today and the collector config carries no logs pipeline; switching events to OTLP logs later is additive (#503 is where that conversation lives).

Samples from the two runtimes are not comparable with each other (cgroup measures the sandbox including sentry overhead; guest-agent measures only the workload's containers) — the source label is what keeps them apart, and rollups must group by it.

Emission cadence

• Periodic: one atelet flag, --actor-stats-poll-interval (0 disables the subsystem). The interval is clamped to a floor of the worst-case sweep — maxActorContainers × statsCallTimeout (25 × 2s) on the micro-VM runtime — with a warning, so a low setting cannot pile overlapping polls onto one guest agent. One knob drives polling, metric updates, and event emission; splitting the cadences is a follow-up if event volume demands it.
• Lifecycle: in addition to the timer, emit an actor-level event at the transitions atelet already owns — a first sample once the workload is up, and a final sample taken immediately before CheckpointWorkload. The final sample is load-bearing twice over:
  • actors that live shorter than the poll interval still leave a usage bracket instead of being invisible to scraping (the suspendable-actor scraping problem Observability At-Scale #174 opened with);
  • cpu_usage_usec restarts at zero on every restore with nothing on the wire marking the epoch — per-epoch final samples let a consumer reconstruct lifetime CPU as the sum of finals plus the live epoch, with no proto change. atelet initiates the checkpoint, so sampling first is free sequencing.

Failure semantics: a periodic sample lost to FAILED_PRECONDITION is skipped silently (mid-boot is a routine state); a failed final sample logs a warning — it is the one sample that can never be retried.

The reader: how atelet finds targets and actor uids

atelet holds no worker→actor mapping today (AteomHerder is deliberately stateless), and GetWorkloadStatsRequest requires the uid it exists to verify. Two candidate designs; this issue should settle the choice:

1. Stateless discovery (preferred): scan ateoms/*/ateom.sock under the /var/lib/ateom-gvisor mount atelet already has, and add a parameterless discovery RPC:

   // GetCurrentActiveWorkloadStats samples whatever this ateom is currently
   // executing, without asserting an identity. Consumers MUST attribute solely
   // from the echoed identity in each sample, never from a mapping they hold.
   rpc GetCurrentActiveWorkloadStats(GetCurrentActiveWorkloadStatsRequest)
       returns (GetCurrentActiveWorkloadStatsResponse) {}
   
   message GetCurrentActiveWorkloadStatsRequest {}
   
   message GetCurrentActiveWorkloadStatsResponse {
     // 0 samples when the ateom is "available" (not an error), 1 when it is
     // "executing" — an ateom serves one actor at a time, and the empty list
     // lets a scraper treat an idle worker as a normal answer rather than an
     // error to classify.
     repeated GetWorkloadStatsResponse stats = 1;
   }

   The scraper needs no prior knowledge and attributes purely from the response echo, so it survives atelet restarts with no recovery path: every ateom creates ateoms/<worker-pod-uid>/ateom.sock when it boots (the same sockets atelet already dials for the lifecycle RPCs), so the first tick after a restart is a readdir plus one probe per socket, and coverage is whole again. The scan needs one tolerance rule — a failed dial means "not a target this tick", never an error — which uniformly covers stale dirs left by deleted worker pods (nothing GCs them eagerly) and workers whose ateom has made its dir but not yet listened. The uid-asserting GetWorkloadStats stays for targeted reads — callers that must be answered about a specific actor, or told via NOT_FOUND that their worker-to-actor mapping is stale.

2. In-memory map: record ateomUID→actorUID from the Run/Restore requests atelet relays, clear on Checkpoint, poll the map. No proto change, but a DaemonSet rollout blinds the reader until each actor's next lifecycle event — long-lived actors go dark for days — unless a recovery path (query ateapi's Worker records, inverting atelet's server-only role) is added, at which point it is not simpler.

Error contract either way: for the discovery RPC, an empty stats list means "available" and is not an error, while FAILED_PRECONDITION still means "executing but no numbers yet — skip this sample". For uid-asserting reads, per the existing doc: NOT_FOUND → the mapping is stale, drop and re-resolve; FAILED_PRECONDITION → keep the target, skip the sample.

Scope

• The atelet poller (flag, clamped interval, socket-scan discovery or map per the decision above).
• GetCurrentActiveWorkloadStats on ateom.Ateom + both runtime implementations, if design 1 is chosen (thin wrappers over the existing attribution + measurement paths).
• Template-level OTel metrics + per-actor OTel log events, as above.
• Lifecycle-triggered first/final samples wired into atelet's existing Run/Restore/Checkpoint paths.
• ate.workerpool.name on the emission path — deferred from Phase 0; atelet knows the pool, so no ateompb change is needed if attribution is stamped at emission time.

Out of scope

• ActorTemplate rollup dashboards, utilization (needs Right-size actor sandboxes to declared ActorTemplate resource limits #679's limits as the denominator), and consumers feat: WorkerPool autoscaling — demand-reactive capacity for warm worker pools #198 / Feature request: suspend actor on inactivity #424 — Phase 2.
• Per-container breakdown for gVisor (sentry accounting) — Phase 3, per Phase 0: StatsWorkload RPC on ateom (proto, both runtime reads, identity retention) #594.
• A wire-level epoch/start-time field for the CPU counter — the final-sample events cover reconstruction; revisit only if a consumer needs epoch detection without the event stream.

Related

• Feature Request: Resource-usage telemetry at actor / ActorTemplate granularity #550 — parent; Phase 0: StatsWorkload RPC on ateom (proto, both runtime reads, identity retention) #594 — Phase 0 (RPC + retention + reads)
• ateom: add GetWorkloadStats RPC and retain actor attribution #667, ateom-gvisor: measure the sandbox cgroup for GetWorkloadStats #739, ateom-microvm: measure the guest agent for GetWorkloadStats #832 — Phase 0 PRs
• Observability At-Scale #174 — cardinality decisions this emission model implements
• Right-size actor sandboxes to declared ActorTemplate resource limits #679 — declared limits; the utilization denominator, Phase 2
• feat: WorkerPool autoscaling — demand-reactive capacity for warm worker pools #198, Feature request: suspend actor on inactivity #424 — consumers of the emitted data