[Design] Actor Otel Telemetry Relay

Design: per-actor otel telemetry relay

Status: draft / for discussion
Related: #761 (actor instrumented metrics issue), #503 (telemetry continuity), #450 (lifecycle hooks)

1. Summary

Actors cannot correctly identify their own telemetry, so all telemetry from an ActorTemplate collapses into a single time series with many writers.

This proposes that ateom expose a fixed OTLP endpoint inside the worker pod, stamp trusted ate.dev/* identity onto everything that arrives.

The principle is one Substrate already applies to logs:

Attribution belongs to the channel, not the payload.

internal/actorlog/logger.go:132-158 already does this for stdout — atelet injects ate.dev/actor_atespace, actor_name, actor_uid,
actor_template_namespace, actor_template_name, container_name into every log line, out of band. The actor never touches them, so they can't be frozen, forged, or lost. Metrics and traces are the signals that haven't caught up.

It's also the same division of labour Kubernetes already uses. Under Prometheus, identity comes from the scrape target, not the workload — instance and podare assigned by the scraper, and emitting them yourself is an anti-pattern.

Actors can't be scraped, so the relay is that job relocated to somewhere it can still happen. This is not a novel mechanism; it's the conventional one adapted to push.

2. Goals

Every actor's telemetry is attributed to that actor, with no cooperation from actor code beyond pointing at an endpoint.
Attribution is trusted — derived by the platform, not asserted by the workload — so it can back per-tenant views, quota, and billing later.
Attribution survives suspend/resume/migration with nothing re-derived.
Storage cardinality is bounded and independent of actor count.
Works on both runtimes, gVisor and micro-VM.

3. Non-goals

Buffered-telemetry loss on onPause: Data. That is OpenTelemetry Application Telemetry Data Loss and Delivery Delays Across Actor Lifecycle Events #503 and needs Actor lifecycle hooks #450's PreSuspend. Complementary, not a substitute.
Control-plane component telemetry, which works today.
Per-actor metric series in long-term storage — explicitly rejected

4. Architecture

┌─ worker pod ────────────────────────────────────────┐
│                                                     │
│  ┌─ actor sandbox ─────────┐      ┌─ ateom ───────┐ │
│  │  app                    │      │  identity     │ │
│  │   │ OTLP                │      │  from the     │ │
│  │   └─► 169.254.17.1:4317 ┼──────┼─►activation   │ │
│  │  (no identity in        │      │  receiver     │ │
│  │   payload)              │      │   · stamps    │ │
│  │                         │      │   · separates │ │
│  └─────────────────────────┘      │   · bounds    │ │
│                                   └───────┬───────┘ │
└───────────────────────────────────────────┼─────────┘
                                            │ OTLP, bounded labels,
                                            │ over a unix socket
                                            ▼
                                   ┌─ atelet (DaemonSet) ─┐
                                   │  relays verbatim,    │
                                   │  one egress per node │
                                   └──────────┬───────────┘
                                              ▼
                                   cluster collector → backend

The actor's exporter config is a constant address, so it survives the golden snapshot intact. What answers at that address is per-actor, re-created on every activation. Substrate already relies on this trick twice: the constant interior IP (cmd/ateom-microvm/net.go:28-30 — "Because that address is a CONSTANT, a restored guest's frozen network config stays valid on any pod") and the constant identity path /run/ate/actor-id.

5. Design

5.1 Transport

169.254.17.1:4317 — the existing worker-side veth gateway, already created per activation by both runtimes.

Chosen over a Unix socket in the identity dir because:

It works on micro-VM. cmd/ateom-microvm/spec.go:91-96 records that the identity bind mount is dropped entirely there — "The micro-VM guest can't see host paths" — so a UDS is unavailable. But kata wires the guest to the same veth via virtio-net (cmd/ateom-microvm/net.go:26-30), so a network endpoint works on both runtimes with no new plumbing.
No host-fd exposure. runsc gates host Unix sockets behind --host-uds, which cmd/ateom-gvisor/runsc.go:84-96 does not pass. Enabling it would widen the sandbox surface and need a threat-model review. A netstack connection needs neither.
Checkpoint safety. An open host fd may block runsc checkpoint; a netstack socket is ordinary severed-connection handling.

The cost is that a shared address relies on one-actor-per-worker for unambiguous attribution. That holds today by definition (docs/glossary.md:30-32) and is the thing to revisit if concurrency lands

5.2 Actor-side configuration

OTEL_EXPORTER_OTLP_ENDPOINT=http://169.254.17.1:4317
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta

Env is safe here because these are constants. The golden-snapshot hazard applies to values that must differ per actor; these do not, so freezing them freezes the right answer.

Note these are captured into the golden, so changing them on an existing template requires a golden rebuild (docs/api-guide.md:101: values are inherited "until the golden snapshot is recreated").

Actors emit no identity attributes. Anything they do emit under ate.dev/is overridden.

5.3 Attribution

ateom sets, overriding any existing value:

service.instance.id              (= actor UID; the standard name for this)
ate.dev/actor_uid
ate.dev/actor_name
ate.dev/actor_atespace
ate.dev/actor_template_namespace
ate.dev/actor_template_name
ate.dev/container_name

That is the set for traces and logs. Metrics get a strict subset (§5.4): actor_uid, actor_name, service.instance.id and actor_atespace are all dropped, because all four are unbounded. The signals still join — on the smaller set — and an exemplar walks from a metric datapoint to a span carrying the rest.

Overriding rather than merging is the point: a workload must not be able to attribute its telemetry to another actor or another atespace.

5.4 Cardinality policy

Signal	Actor identity	Rationale
Metrics	stripped before forwarding	a series is durable: index entry, interned labels, chunks, retained for the full window after the actor dies
Traces	kept	a span is an event, not a series — 10k actors/day is 10k rows, not 200k index entries
Logs	kept	already the case today

Forwarded metric labels: actor_template_namespace, actor_template_name, container_name. Output series count is templates × containers × metrics — constant, and independent of how many actors ever ran.

5.5 Forwarding and enrichment

Forwarded telemetry must not be re-enriched with k8s.pod.name / k8s.node.name upstream. Those describe the current host, change on every migration, and are what makes the failure read like ordinary pod churn. ateom should stamp ate.dev/worker_pod / ate.dev/worker_node instead — clearly named as host facts, not workload identity — and keep them off metric resources entirely.

ateom does not dial the cluster collector. It forwards over a node-scoped unix socket served by atelet, which holds the single upstream connection for the node. A direct path requires the worker pod to have network egress to the collector, and the worker pod is the pod running untrusted actor code. Forwarding over a unix socket on a host directory the pod already mounts means the worker pod can be denied network egress entirely and telemetry still flows. No configuration of a direct path yields that property.

Related: #809

5.6 Suspend and resume

Nothing to re-derive. The frozen exporter config points at a constant address; on resume that address is served by the new worker's ateom, which knows the new activation. The gRPC client sees a dead connection and reconnects to the same address, which now leads somewhere else. That is the whole mechanism.

This is why attribution does not depend on #450, though #450 is still needed for flushing (#503).

5.7 Temporality

Delta is required, not preferred.

Under cumulative, ateom would have to hold each actor's last-seen value and compute the merged total itself. On migration the destination ateom has no history and the merged series drops — the original bug, one layer up. Under delta each export is self-contained, so contributions from different workers simply add.

Delta also cancels the golden baseline. The SDK's last-collection bookmark is snapshotted alongside the counter, so a restored actor reports only its own work.

Delta is genuinely lossier. Cumulative self-heals across a dropped export; delta does not, so an onPause: Data kill loses that window permanently. Still the right trade, because what cumulative preserves faithfully is a wrong number.

6. Open questions

Concurrency. A shared worker-pod address relies on one-actor-per-worker. If that changes, attribution needs per-actor addressing or a per-actor socket. Worth knowing the cost before it's forced.
Gauges: Delta temporality does not apply to gauges. Two actors of one template writing the same gauge now share one series and race on last value. ateom cannot fix this. Maybe we should just make it explicit that per-actor (instance) gauges do not work for substrate and workload should emit logs/events when per-actor gauges are needed, and prefer counters and histograms wherever the quantity is additive.
Actor telemetry shares ateom's channel to atelet: Identities do not get mixed up, because atelet forwards messages unchanged and OTLP keeps a separate resource block per sender. The problems are elsewhere. First, if the socket is missing, ateom: export telemetry through an atelet unix-socket OTLP relay #809 falls back to sending over the pod network; that is right for ateom's own data but wrong for actor data, since this design uses the socket so the worker pod needs no network at all. Second, one connection per node carries both kinds, so a noisy actor can delay Substrate's own telemetry. Suggested order: stop the fallback for actor telemetry and rate-limit it in ateom now, then give atelet a second connection to the collector, and only later give actor telemetry its own socket — and only when atelet itself must treat the two kinds differently. Note that a separate socket only tells the two kinds apart; it does not prove who sent the data unless the file permissions change too.

Krisztian F (@krisztianfekete)

cc Krisztian F (@krisztianfekete) Tim Bai (@baizhenyu) Jeff Luo (@JeffLuoo)

Overall the concept LGTMs, just some comments we need to follow up on:

169.254.17.1:4317 won't work in the proposal. The nftables redirect intercepts the OTLP connection before ateom sees it.

ActorEgressRedirectRule (internal/ateomnet/net.go:360-373) matches the source IP 169.254.17.2 and TCP. It has no destination match. The table goes into the worker pod netns, because net.go:571 is outside the NetNSDo at net.go:564, and the chain hooks prerouting. As the result, an actor connection to 169.254.17.1:4317 becomes a connection to 169.254.17.1:<atunnel egress port>. The receiver gets nothing.

The failure is also conditional. The redirect exists only when egress_gateway is set (net.go:363-365, ateom.proto:99-100). An actor with direct egress works. An actor with tunneled egress fails.

We may need to discuss it with people who work on networking layer to review this section and provide guidance on the section.

container_name works for logs because each container has its own pipeline using WrapContainerLogs in the logger. If we require users to use OTLP in their telemetry pipeline for containers in actor, it has no equivalent so we need to let users configure to set up container name from application layer.

Bowei Du (@bowei)

Overall the concept LGTMs, just some comments we need to follow up on:

169.254.17.1:4317 won't work in the proposal. The nftables redirect intercepts the OTLP connection before ateom sees it.

ActorEgressRedirectRule (internal/ateomnet/net.go:360-373) matches the source IP 169.254.17.2 and TCP. It has no destination match. The table goes into the worker pod netns, because net.go:571 is outside the NetNSDo at net.go:564, and the chain hooks prerouting. As the result, an actor connection to 169.254.17.1:4317 becomes a connection to 169.254.17.1:<atunnel egress port>. The receiver gets nothing.

The failure is also conditional. The redirect exists only when egress_gateway is set (net.go:363-365, ateom.proto:99-100). An actor with direct egress works. An actor with tunneled egress fails.

We may need to discuss it with people who work on networking layer to review this section and provide guidance on the section.

container_name works for logs because each container has its own pipeline using WrapContainerLogs in the logger. If we require users to use OTLP in their telemetry pipeline for containers in actor, it has no equivalent so we need to let users configure to set up container name from application layer.

Bowei Du (@bowei) Lior Lieberman (@LiorLieberman) haiyanmeng Can you provide suggestion as the network team: under tunneled egress, is 169.254.17.1 supposed to be reachable from the actor at all, or is the gateway address treated as "outside" like everything else?

Da Huang (@git286)

Thanks Da Huang (@git286) for putting this together! Adding some comments/questions in-line below (it's very challenging to do this in GH issues, maybe we can move this to Google Docs?):

It's also the same division of labour Kubernetes already uses. Under Prometheus, identity comes from the scrape target, not the workload — instance and pod are assigned by the scraper, and emitting them yourself is an anti-pattern.

The reason per-target series are fine under Prometheus is not just that the scraper assigns identity, it is that instance is the writer identity. Relocating the scraper's job means relocating that part too.

This proposes that ateom expose a fixed OTLP endpoint inside the worker pod, stamp trusted ate.dev/* identity onto everything that arrives.

ate.dev/* is the wrong namespace, we should use internal/ateattr.

Attribution is trusted — derived by the platform, not asserted by the workload — so it can back per-tenant views, quota, and billing later.

Why are we dropping actor_atespace in 5.3 then? Quota and billing cannot work with sampled data. Tenants are not actors; cardinality here is tenant count, which is what we'd need for multi-tenancy, which is in scope, right? Or, usage is calculated as unsampled events like we discussed in #174.

It works on micro-VM. cmd/ateom-microvm/spec.go:91-96 records that the identity bind mount is dropped entirely there — "The micro-VM guest can't see host paths" — so a UDS is unavailable. But kata wires the guest to the same veth via virtio-net (cmd/ateom-microvm/net.go:26-30), so a network endpoint works on both runtimes with no new plumbing.

I've been checking this as part of another issue and think it should work!

Chosen over a Unix socket in the identity dir because:

We should serve 4318/http-protobuf as well as 4317/grpc. The OTel spec's default protocol is http/protobuf, several SDKs ignore or only partially implement OTEL_EXPORTER_OTLP_PROTOCOL, and an actor that defaults to 4318 gets a silent black hole with no error anywhere. Every collector should serve both.

OTEL_EXPORTER_OTLP_ENDPOINT=http://169.254.17.1:4317
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta

Let's also add OTEL_METRIC_EXPORT_INTERVAL. SDK default is 60s, so an actor that lives for shorter periods cannot export and that's very common in this project. We already hit this in #634.

I'd also add OTEL_SERVICE_NAME, or better, default it at the relay. Without it the SDK emits unknown_service: and every actor of every template collapses into one service. Defaulting at the relay is probably better here because it is then fixable without a golden rebuild.

Note that the temporality preference is a request, not an enforcement. A workload can ignore it and some SDKs do not implement it unfortuntately, so the relay must handle cumulative arrivals regardless.

Overriding rather than merging is the point: a workload must not be able to attribute its telemetry to another actor or another atespace.

This might deserve its own design, but roughly we probably have to

do this for every attribute, not just the resource. Resource, scope, span, span event, span link, datapoint, log record as a forged ate.actor.uid on a datapoint can work around a resource-level override, and if we strip actor identity from metrics it becomes the only identity attribute reaching storage. Exemplars should also have to be included, once we have them.
remove everything under ate.* unconditionally at every phase, then set the trusted values. Same goes for for k8s.* as well I guess,
think about some mehanism to make forging very hard. Telemetry identity is the service.namespace + service.name + service.instance.id, so we can have ateom set service.namespace as a trusted override probably.
have a trusted resource attribute (ate.telemetry.plane=actor, or similar) for routing. E.g. for the resource_attribute for k8sattributes' pod_association, rate-limiting, other use-cases?

Metrics get a strict subset (§5.4): actor_uid, actor_name, service.instance.id and actor_atespace are all dropped, because all four are unbounded. The signals still join — on the smaller set — and an exemplar walks from a metric datapoint to a span carrying the rest.

See my comments above for atespace. Also, joining via exemplars can only work if the actor also sets them. These are fine limitatations, but we have document them, and choose the approach we want for usage use-case.

Forwarded metric labels: actor_template_namespace, actor_template_name, container_name. Output series count is templates × containers × metrics — constant, and independent of how many actors ever ran.

This is a multi-writer otlp stream that is an anti-pattern because we won't be able to distinguish the emitters.
I'd add k8s.pod.name (+ probably k8s.pod.uid) of the worker pod. This way we have single writers only and cardinality is worker pods not actors, which should be fine.

We also could ship everything here, and do the plumbing/trimming in an OTel collector, but I'd do this once we hit scaling limits with the worker approach above.

Forwarded telemetry must not be re-enriched with k8s.pod.name / k8s.node.name upstream. Those describe the current host, change on every migration, and are what makes the failure read like ordinary pod churn. ateom should stamp ate.dev/worker_pod / ate.dev/worker_node instead — clearly named as host facts, not workload identity — and keep them off metric resources entirely.

k8s.pod.name and k8s.node.name are the semconv registry attributes for these concepts, we can likely use k8sattributes processor via resource_attribute-based pod_association rule. See my comment above about getting rid of these.

Delta also cancels the golden baseline. The SDK's last-collection bookmark is snapshotted alongside the counter, so a restored actor reports only its own work.

Not sure if this is true, I think start time will be also in golden. We probably have to rewrite it in the relay.

Open questions

Concurrency

Agreed, we can revisit it later.

Gauges

+1, with wo additions:

asynchronous UpDownCounters can be delta and backends handle them unevenly, so name them too
if the relay ever aggregates (see my point above as an alternative), gauges are partially recoverable with min/max/avg rather than last-value-wins. Worth making this a stated limitation in the docs, not just an open question, because it is a real constraint on what workloads can instrument.

Actor telemetry shares ateom's channel to atelet; on the ordering part

I'd be happy if ateom rate limit and the fallback removal would land with the feature, so I would move both out of open questions and into the design.

container_name works for logs because each container has its own pipeline using WrapContainerLogs in the logger. If we require users to use OTLP in their telemetry pipeline for containers in actor, it has no equivalent so we need to let users configure to set up container name from application layer.

I think we can auto-set this via something like OTEL_RESOURCE_ATTRIBUTES=container.name=<name>, if we have access to the name in our OCI layer.

Zoe Zhao (@zoez7)

maybe we can move this to Google Docs

SG. The goal of this initial draft is to make agreement on the high level direction: actor -> ateom -> atelet -> collector relay + attribution, network etc. Will leave some details in the doc and implementation.

ate.dev/* is the wrong namespace, we should use internal/ateattr.

Thanks for catching that! Agreed we should use the ateattr.

Why are we dropping actor_atespace in 5.3 then

I think Zoe Zhao (@zoez7) mention before that the cardinality of atespaces is closer to that of actors

We should serve 4318/http-protobuf as well as 4317/grpc. The OTel spec's default protocol is http/protobuf, several SDKs ignore or only partially implement OTEL_EXPORTER_OTLP_PROTOCOL, and an actor that defaults to 4318 gets a silent black hole with no error anywhere. Every collector should serve both.

Sure.

Let's also add OTEL_METRIC_EXPORT_INTERVAL. I'd also add OTEL_SERVICE_NAME, or better, default it at the relay.

SG

I'd add k8s.pod.name (+ probably k8s.pod.uid) of the worker pod. This way we have single writers only and cardinality is worker pods not actors, which should be fine.

yeah, I think I missed that

Sounds good, can you please start the design doc when you have a chance?

Also opened this one: #1036

(chore): consolidate logging to ateattr (#1035)

Benjamin Elder (@BenTheElder)

Benjamin Elder (@BenTheElder) said the one-actor-per-worker model might change soon. Will hold this for now to wait for more information on the system reachitecture

That is still exploratory but I do intend to propose something ~next week, wasn't quite ready in time for the community call this week.
Apologies.

Benjamin Elder (@BenTheElder)

Sounds good Benjamin Elder (@BenTheElder), please share it if it's available before the community call this week!

Zheng Lammerts (@zlammerts-svg)

I just added it to the P0/M3 milestone for now.

Zheng Lammerts (@zlammerts-svg) FYI

Da Huang (@git286), No worries, thanks for the update! I'd also say this is P0, but just noticed that the label applied is for P1. The phrasing of the comment also makes me think that you might wanted to say that this is P1 instead of P0. Which one it is? :)

Sorry for the confusion and it was a typo! I actually mean we decided that it will NOT be a P0 blocking issue for GA.

In my opinion this is absolutely P0, if we are going to merge multi-actor workers, we need a way to properly observe them

FWIW, this is already a operational gap even without multi-actor support.

Eitan Yarmush (@EItanya)

Eitan Yarmush (@EItanya) Krisztian F (@krisztianfekete) thanks for pushing on this, and sorry for the confusion my typo caused. Before we settle on a label, I'd like to step back to the problems rather than the solution, because the relay is one mechanism that happens to solve several different things, and I don't think they are all P0.

The way I see it the relay addresses four separate problems:

Network path. Actor telemetry today needs a route from the worker pod to the collector. The relay makes that path a unix socket so the collector is never reachable from inside the sandbox.
Trusted identity. Whatever identity is on a span or metric today was put there by the actor. The relay makes the platform set it, so it can back per-tenant views, quota, and usage later.
Single writer. All actors of a template land on the same metric series with many concurrent writers, and the backend can't separate them. Multi-actor makes this worse.
Golden state. The SDK's instance id, counter baseline, and start time are frozen in the golden snapshot, so every clone reports the same identity and inherited counts.

Our reasoning for P1 was that for GA, most of these can be mitigated on the actor side without any platform change:

1 is already policed by the egress tunnel like any other actor traffic. Whether the collector is reachable is an egress policy decision the operator already controls.
2 is the one that genuinely cannot be done on the actor side, since trust by definition can't come from the workload. But nothing in GA scope consumes trusted attribution yet. Quota and billing are post-GA.
4 goes away if the workload initializes its OTel SDK after activation instead of at image build, or resets it in a restore hook once Actor lifecycle hooks #450 lands. It can read its identity from /run/ate/actor-id, which is regenerated per activation. That gives a clean counter baseline and a unique instance id. We'd document this as a template authoring rule.
3 can be partly mitigated on the actor side by using delta etc

My question back is: which of these do you see as the P0 gap that must be solved in the substrate layer for GA? I'd also genuinely like to understand what you are running into on your end, since that is the best input for deciding what is blocking.

Thanks for pushing back, these are all great points, maybe we should discuss this at the community meeting today and see if we can come to a conclusion?

Thanks for pushing back, these are all great points, maybe we should discuss this at the community meeting today and see if we can come to a conclusion?

Sounds good!

Da Huang (@git286)

Thanks Da Huang (@git286), here are my notes as I can't make the call today unfortunately:

I am fine-ish having 2 as P1, but 3 and 4 are P0 already, with one actor per worker. If an actor sets service.instance.id from its own identity, every actor gets its own series and that's a cardinality explosion. If it doesn't, all actors of a template write to one series at once. Without the relay, every template's metrics are wrong by default, and operators can't even see it. Once we have multi-actors, it will be even worse.

How about making the minimal relay P0? ateom sets service.instance.id to the pod, rewrites timestamps, and forwards over the atelet socket.

Krisztian F (@krisztianfekete)

Thanks Da Huang (@git286), here are my notes as I can't make the call today unfortunately:

I am fine-ish having 2 as P1, but 3 and 4 are P0 already, with one actor per worker. If an actor sets service.instance.id from its own identity, every actor gets its own series and that's a cardinality explosion. If it doesn't, all actors of a template write to one series at once. Without the relay, every template's metrics are wrong by default, and operators can't even see it. Once we have multi-actors, it will be even worse.

How about making the minimal relay P0? ateom sets service.instance.id to the pod, rewrites timestamps, and forwards over the atelet socket.

Krisztian F (@krisztianfekete) agreed that 3 and 4 are the more important ones of the four. However, I'm still not convinced is that they need to BLOCK GA. To be clear about what P1 means here: we are not stopping work on this. The design doc is nearly ready and the relay stays the plan. It becomes a stretch goal for M3 rather than a gate, which means it may land after the freeze if the P0s take the time.

For 3, there is a quicker mitigation we can ship in the meantime. The SystemInfo volume already projects actor metadata files per activation (name, atespace, uid). We can add the worker pod UID and, once #1266 lands, the slot number. An actor that builds its OTel resource from those files gets a bounded, single-writer instance id with no relay involved. The honest gap is migration: the SDK resource is fixed at creation, so a resumed actor keeps stamping its old pod until it can rebuild its provider. That is what the lifecycle hooks in #450 are for, a restore hook that refreshes identity before the actor resumes. So the mitigation is "identity files now, correct across migration once #450 lands", and it needs the template author to read the files after activation rather than at build. Not a guarantee, but a real improvement over today for anyone who does it.

One question that would help me calibrate: is this causing real problems in a running deployment on your side today, or is it that we both expect it to once actor counts grow? If someone is already looking at wrong dashboards, that changes the priority conversation. If it's anticipated, I'd rather spend the pre-freeze weeks on the things already breaking and land the relay right after.

Eitan Yarmush (@EItanya)

Quick summary of the discussion with Eitan Yarmush (@EItanya) in today's community meeting, for those who couldn't make it (Krisztian F (@krisztianfekete)):

Not a GA blocker. We agreed this does not need to BLOCK GA if it is not done by the freeze, and "blocks GA" is what P0 means for us. So the label stays P1 and the issue stays in M3 as a stretch goal.
Still important, still planned. This is not being dropped or deferred indefinitely. We continue on it as soon as the P0s are landed, and the design doc will go out before then so the technical questions on the thread get answered in parallel.
Order of work. When implementation starts, the first slice is the part that fixes 3 (single writer) and 4 (golden state): ateom serving the OTLP endpoint, setting service.instance.id to the worker pod, rewriting timestamps, and forwarding over the atelet socket.

Krisztian F (@krisztianfekete) this is close to the minimal relay you proposed, just sequenced behind the P0s rather than made one. Please shout if you think the sequencing misses something, and the question from my earlier comment still stands: if this is already hurting a real deployment/user, that is the strongest input for pulling it forward.

Removing from M3 blocker as a result of conversation

Just checked the multi-actor networking work and I think the networking model from #1689 makes the relay's network part easier.

Every actor has its own network namespace, named by actor UID. The gateway address 169.254.17.1 lives inside that per-actor namespace, and the actor still sees the same fixed guest address as before, so frozen exporter configs stay valid. ateom already serves two per-actor services from sockets it opens inside that namespace: the atunnel egress listener and the DNS relay. It opens the socket with a thread switched into the namespace, binds the handler to the actor UID before serving, and serves from the pod namespace. The actor has no path to the pod interface at all; every outbound TCP connection is redirected to atunnel, and the redirect excludes the sandbox's own /30 so the gateway's own listeners are reachable.
Endpoint. The relay is the third per-actor service on the gateway address. ateom opens listeners on port 4317 (gRPC) and 4318 (HTTP/protobuf) inside each actor's gateway namespace, using the same helper atunnel uses, and serves them from the pod namespace. Both ports are served because the OpenTelemetry default protocol is HTTP/protobuf and several SDKs do not honor the protocol setting. This works identically on gVisor and micro-VM, since both have a gateway namespace.
Telling actors apart. The listener is the identity. A socket inside actor A's namespace can only be reached by actor A, so ateom knows the sender the moment a connection is accepted, with no lookup. No source-address mapping, no per-slot port, no NAT. Each actor also has its own socket table, so two actors dialing from the same guest address and ephemeral port cannot collide.
No nftables change. The OTLP endpoint sits on the gateway address, and the existing redirect already leaves traffic to the sandbox /30 alone. The relay installs no rules and needs no exception.
Connection lifetime. ateom binds the OTLP handler to the actor before the listener starts serving, following the atunnel pattern, so an accepted connection can never pick up a later activation's identity. On teardown the sandbox session closes its sockets and deletes the namespaces, which cuts off anything the old actor could still send. ateom must also stop its gRPC and HTTP servers for that actor at session close, since closing a listener does not close connections already accepted.
Why not a unix socket. A network endpoint on the gateway address works on both runtimes with the plumbing that exists. A unix socket in the identity directory does not exist on micro-VM, would need host unix sockets enabled in gVisor, and adds an open host fd that can block checkpoint.
Multi-actor. Nothing here changes when a worker holds more than one actor. Each actor gets its own namespace and its own OTLP listeners. The only multi-actor-specific input the relay needs is the slot number to stamp, which is a value, not a mechanism.

Google doc version:
https://docs.google.com/document/d/1jXDUuteyXJ_H2AcYtPA_PG8FMvjMrQbGqkv2qiNXmAk/

Navigation Menu

Repository navigation

[Design] Actor Otel Telemetry Relay #853

Design: per-actor otel telemetry relay

1. Summary

2. Goals

3. Non-goals

4. Architecture

5. Design

5.1 Transport

5.2 Actor-side configuration

5.3 Attribution

5.4 Cardinality policy

5.5 Forwarding and enrichment

5.6 Suspend and resume

5.7 Temporality

6. Open questions

Open questions

10 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Design] Actor Otel Telemetry Relay #853

Description

Design: per-actor otel telemetry relay

1. Summary

2. Goals

3. Non-goals

4. Architecture

5. Design

5.1 Transport

5.2 Actor-side configuration

5.3 Attribution

5.4 Cardinality policy

5.5 Forwarding and enrichment

5.6 Suspend and resume

5.7 Temporality

6. Open questions

Activity

git286 commented on Aug 11, 2026

JeffLuoo commented on Aug 11, 2026

git286 commented on Aug 11, 2026

krisztianfekete commented on Aug 12, 2026

Open questions

git286 commented on Aug 14, 2026

krisztianfekete commented on Aug 17, 2026

krisztianfekete commented on Aug 18, 2026

git286 commented on Aug 20, 2026

BenTheElder commented on Aug 21, 2026

krisztianfekete commented on Aug 25, 2026

10 remaining items

EItanya commented on Sep 24, 2026

git286 commented on Sep 24, 2026

EItanya commented on Sep 24, 2026

krisztianfekete commented on Sep 24, 2026

git286 commented on Sep 24, 2026

EItanya commented on Sep 24, 2026

git286 commented on Sep 24, 2026

krisztianfekete commented on Sep 24, 2026

git286 commented on Sep 24, 2026

git286 commented on Sep 24, 2026

EItanya commented on Sep 24, 2026

git286 commented on Sep 25, 2026

git286 commented on Oct 5, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions