Repository navigation
Initial set of platform metrics around lifecycle, scheduling, and fleet-saturation #433
Description
Opening this issue to track the foundational work around performance/scaling-oriented platform metrics that came up in #174 and in the load-testing threads (#168, #176, #98, #44). I proposing a high-level plan here first, but if we get too deep in the details (exact attributes, buckets, etc.), happy to open a Google Doc so we can discuss it there as well.
Scope
When we load test substrate today, we cannot really answer core operational questions with today's telemetry, e.g. is the fleet healthy under load, and where is the bottleneck?
We have three metrics total (rpc.server.call.duration, atenet.router.route.duration, atelet.snapshot.size), but nothing for lifecycle operations, pool saturation, or scheduling outcomes, and some components (ateom, atecontroller) emit no metrics at all. I think we have a good foundation in #174, so I am cleaning that up and using this to track the implementation.
- otlp push instead of pull
- Cardinality-wise actor, atespace, and session atrs go on traces/logs/events only, never on metric labels. Template, worker pool, sandbox class, outcome/phase, and cluster are the metric dimensions. We can add exemplars on the duration histograms, so a latency spike can be tracked back to a sampled actor trace. This can be a follow-up.
- Actor level analytics will be covered by events (named LogRecords in OTel). This came up in Observability At-Scale #174's "are there vital metrics we should push out to logs". It needs the cgroup work first (fix(atelet): set OCI CgroupsPath so each actor gets its own cgroup #161, blocked on All actors are under the
pausecgroup #288), so just mentioning this here for context for now
Initial set of metrics
ateapi emits the lifecycle/scheduler/fleet signals, atelet emits the restore/checkpoint phase information.
| Metric | Type | Use-case |
|---|---|---|
ate.actor.lifecycle.operation.duration (operation.name, error.type, template, pool, sandbox class, snapshot kind) |
histogram | activation-latency, suspend/resume rates and error rates |
ate.actors (status, template) |
updowncounter | fleet density, actor-to-worker multiplexing ratio |
ate.workerpool.workers (pool, state, sandbox class) |
observable updowncounter | pool saturation (see #198) |
ate.scheduler.assignment.duration (pool, outcome, error.type) |
histogram | "no free workers" rate (see #27, #398) |
ate.actor.restore.duration / ate.actor.checkpoint.duration (phase, template, sandbox class, snapshot kind) |
histograms | cold-start bottleneck, see related #166/#228 |
cc Alex Van Boxel (@alexvanboxel) Julian Gutierrez Oschmann (@juli4n), Zoe Zhao (@zoez7), Eitan Yarmush (@EItanya)
Activity
krisztianfekete commentedon Jul 13, 2026
Started to flesh these out in #434. We can break that into smaller chunks, if we want, just wanted to test all holistically locally as I've been iterating on the proposal.
juli4n commentedon Jul 20, 2026
Thanks, Krisztian F (@krisztianfekete).
I think separating this into different PRs will very likely expedite the review process.
krisztianfekete commentedon Jul 20, 2026
Hi Julian Gutierrez Oschmann (@juli4n), sure, happy to; I just started out with one PR as I've been testing all these holistically. What's your preference for splitting?
juli4n commentedon Jul 21, 2026
How about we split it in the way you described it in the table? Sound like sufficiently independent changes?
krisztianfekete commentedon Jul 22, 2026
I'll probably do something like this
feat(otel): add ate.workerpool.workers metric with e2e suite (#492)
feat(otel): add actor lifecycle + scheduler duration metrics (#514)
feat(otel): add cold start metrics (#776)
krisztianfekete commentedon Aug 7, 2026
This is now done.
krisztianfekete commentedon Aug 7, 2026
Reopening for
ate.actors, which will be a sub-issue and needs design first.