Observability At-Scale

If we are load-testing a system, we should also be able to validate that the system is functioning appropriately. Observing metrics from the load-source is well-understood, observing the behaviors of Substrate are not:

Actors can be suspended which makes prometheus-style scraping a problem (push-based may help here if we flush metrics before sleep, but that interferes with batching/sampling)
how do we attribute resource consumption to an individual actor? Do we need to?
Can most benchmarking metrics be satisfied via scraping atelets/API server to validate things like # of successful suspend/resume?

Logs are already persistent, are there vital metrics we should push out to logs? Do we want an easy way of re-aggregating these metrics.

Example uses:

Validate total # of bytes written/overwritten (or FDs opened or network requests, etc.)
Aggregate non-fatal error counts (e.g. a long-running process attempts to poll an endpoint but times out)
Observe latency of requests as seen by actor
Aggregate request statistics when agent is acting as a server

I'm willing to help here. I'm a contributor to OpenTelemetry, and I think this could be up my alley: making sure this project is OTel first, and I see in the long-term bringing some of the discoveries we're making back into the OpenTelemetry conventions.

I don't know if we want to be scraping apiserver/atelet as leaving bandwidth for critical path might be needed in load spikes. An advantage of using the DB layer to track via Debezium or PubSub. Sudo CDC. Though that would tie the observability layer to the underlying DB solution and if that is pluggable, the observation layer would need to be equally pluggable.

Alex Van Boxel (@alexvanboxel)

Alex Van Boxel (@alexvanboxel) Would love to work with people on figuring out a solution for this.

I definitely think there is something to scraping a local FD rather than using a live endpoint -- seems like it might make observing suspended actors easier. That seems roughly analogous to Debezium?

Grant McCloskey (@MushuEE), I'm sensitive to not wanting to hog bandwidth, but there will always be some kind of resource drain associated with monitoring. Are you thinking that push-based models would be a better fit under load spikes?

I think If we look at the dimension "where is the metric emitted from" there are 2 types.

Metrics captured by the agent workload (for example, latency of requests as observed by the actor)
Actor metrics captured by ate system (for example, CPU and memory usage, number of successful suspend and resumes)

If we look at the dimension "what is the granularity", there are several possibilities:

Per-actor granularity - the K8s analogy is the per-container metrics. ~1 billion per cluster. Having such metrics can be useful when debugging certain actor's OOM issues for example.
Per-atespace granularity - the Agent Developer layer ultimately decides the cardinality of atespaces. Upper bound is still ~1 billion per cluster. This could be useful to prevent the noisy neighbour problem, and configure quotas per atespace.
Per-actorTemplate level granularity - this would be similar to "per application", however, when the user performs binary upgrade of the same "application", current plan is to create a new ActorTemplate, therefore, the "Per-actorTemplate level granularity" is similar to "per application per version". This could be high cardinality as well, depending on how users define it. So far we think this would be ~n*1000 per cluster.
Per-cluster level.

Also thinking about the target users of theses metrics. This issue was opened with the motivation of benchmarking (i.e. when we are load testing the system, how do we know the actor is functional, as well as explaining any bottlenecks we run into. But in practice the same metrics are used by Agent Developers to operate their agent workloads.

Some user stories:

As an agent developer, when I deploy an ActorTemplate, I should be able to export metrics using a standardized framework, same as deploying a regular k8s workload.
As an agent developer, I should be able to drill-down per-actor metrics during debugging. (for example "why is a particular instance crashing -> oh OOM") - But this can rely on structured logging by ateom as well.
As an agent developer, I should be able to configure per-atespace quota. (for example, num_cpu_per_atespace would be useful? if the agent developer configures per-user atespaces)
As an substrate developer, I should be able to observe the profile workloads are functioning properly under load.

Tentative decisions:

The "actor" and "atespace" cardinalities should stay in logs and traces, but not as a metric label in a TSDB.
"ActorTemplate" and "Cluster" should exist as metric labels in a TSDB. (With the assumption that the number of "ActorTemplate" defined by users stays small i.e. ~thousands)
For configuring per-atespace quota (i.e. the 3rd user stories above), we should follow kubernetes and use requested resource instead of real time usage.

smythe

Discussed with smythe offline, for alpha benchmarking, observing the API is sufficient. Moved this one out of the alpha milestone.

Also getting per-actor CPU and memory usage is dependent on fixing #288

atelet: poll ateom workload stats into template-level metrics

atelet: poll ateom workload stats into template-level metrics (#961)

atelet: emit per-actor usage events from the stats sweep (#1206)

Navigation Menu

Repository navigation

Observability At-Scale #174

12 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Observability At-Scale #174

Description

Activity

alexvanboxel commented on Jun 4, 2026

MushuEE commented on Jun 4, 2026

maxsmythe commented on Jun 5, 2026

zoez7 commented on Jun 21, 2026

zoez7 commented on Jun 21, 2026

zoez7 commented on Jun 22, 2026

zoez7 commented on Jun 22, 2026

zoez7 commented on Jun 23, 2026

zoez7 commented on Jun 23, 2026

12 remaining items

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions