Repository navigation
OpenTelemetry Application Telemetry Data Loss and Delivery Delays Across Actor Lifecycle Events #503
Description
1. Issue Background & Context
User applications running as Agent Substrate actors frequently emit custom metrics and distributed trace spans using OpenTelemetry (OTel) push exporters (otlpmetricgrpc, otlptracegrpc, or otlpmetrichttp).
In typical production workloads:
- OTel Push Exporter Interval: e.g., 60 seconds (standard batch export timer).
- Active Execution Duration Before Idle Suspension: e.g., 30 seconds (active request handling window before an idle timeout or on-demand pause occurs).
When the push exporter interval (e.g., 60s) is longer than the active execution duration before suspension (e.g., 30s), telemetry events generated during the active phase may remain buffered in the OTel SDK's in-memory queues (PeriodicReader accumulators and BatchSpanProcessor ring buffers).
This issue summarizes the core challenges, empirical findings across Substrate snapshot modes, short-term mitigations for actor authors, and proposed long-term system-level solutions.
2. Identified Telemetry Issues Across Actor Lifecycle Events
Through architectural analysis and lifecycle benchmarking, five primary telemetry issues were identified:
Issue 1: In-RAM Telemetry Loss in Data Snapshot Mode (onPause: Data) — HIGH
- Root Cause: When
snapshotsConfig.onPauseis set toData, Substrate excludes process memory (RAM) from the snapshot and saves only attachedDurableDirdisk volumes. The container process is killed upon pause. - Impact: 100% loss of pre-pause in-memory telemetry. All metrics and trace spans generated during the 30s before pause that were awaiting the 60s push timer are permanently lost when the process is killed.
Issue 2: Telemetry Abandonment for Un-Resumed Suspended Actors — HIGH
- Root Cause: In Full Snapshot Mode (
onPause: Full), process RAM is saved to snapshot storage. However, if an actor is suspended and subsequently deleted or garbage-collected (kubectl ate delete actor) without ever being resumed, the buffered in-memory metrics remain stuck in the snapshot file. - Impact: Un-pushed telemetry is never delivered to the OTel Collector.
Issue 3: Delivery Delays & Wall-Clock Timer Expiration — MEDIUM
-
Root Cause: In Full Snapshot Mode (
onPause: Full), when an actor is suspended for 5 minutes (from$t = 30\text{s}$ to$t = 330\text{s}$ ), wall-clock time advances on the host machine. -
Behavior: Upon resume at
$t = 330\text{s}$ , the language runtime (Go/Python/Node) evaluates the 60s timer deadline ($D = 60\text{s}$ ). Because$330\text{s} > 60\text{s}$ , the timer expires immediately upon resume at$t = 330\text{s}$ , flushing all pre-suspend and post-resume telemetry in a single batch. -
Impact: Delivery to the OTel backend is delayed by the suspension duration, though original generation timestamps (
$t = 10\text{s}$ ) remain accurate.
Issue 4: Backend Retention Window Expiration & Stale Timestamp Rejection — HIGH
- Root Cause: If an actor remains suspended for hours or days before being resumed, when it wakes up, it flushes telemetry with timestamps from hours/days ago.
- Impact: Telemetry backends (Prometheus, Cloud Monitoring, Datadog) drop incoming samples older than their allowed ingestion window (e.g., Prometheus drops samples > 2 hours old).
Issue 5: Telemetry Loss on Sudden Actor Termination (Issue #23) — HIGH
- Root Cause: When an active or suspended actor is explicitly terminated (e.g., via
kubectl ate delete actor, container eviction, or TTL cleanup), Substrate sends aSIGTERM/SIGKILLto the workload container. - Impact: If the application process does not catch
SIGTERMand callmeterProvider.ForceFlush(ctx)/tracerProvider.ForceFlush(ctx)during the graceful termination period (or if killed abruptly viaSIGKILL), 100% of un-flushed in-memory metrics and spans are lost upon container destruction.
3. Empirical Test Matrix
Empirical test suite results across lifecycle scenarios:
| Test Scenario | Snapshot Mode (snapshotsConfig) |
In-Memory Telemetry Loss? | Delivery Delay | Socket Recovery | Test Results |
|---|---|---|---|---|---|
| Suspend & Resume | onPause: Full |
~0% Loss under standard resume conditions | Delayed by suspend duration; fires immediately at resume ( |
Automatic re-dial | Telemetry in RAM preserved across snapshot; delivery delayed by suspension duration until immediate post-resume timer flush. |
| Pod Migration | onPause: Full |
~0% Loss under standard resume conditions | Delayed by migration duration | Automatic re-dial | Telemetry in RAM preserved; OTLP gRPC/HTTP exporter automatically re-dials new worker pod IP post-migration. |
| Suspend & Resume | onPause: Data |
100% Loss of pre-pause buffer | N/A (Data lost on process kill) | N/A | 100% loss of pre-pause in-memory telemetry because process RAM is excluded from snapshot and container is killed. |
| Data Mode + Disk Replay |
onPause: Data + DurableDir
|
~0% Loss (Persisted to disk) | Flushed post-resume from disk | Automatic re-dial | 0% loss; telemetry persisted to DurableDir volume survives process kill and is replayed to collector upon startup. |
| Actor Termination | Any (Full or Data) |
100% Loss (unless SIGTERM handler flushes) | N/A (Container destroyed) | N/A | 100% loss of buffered in-memory telemetry upon container shutdown unless application handles SIGTERM and invokes ForceFlush() before exit. |
4. Short-Term Mitigations (For Actor Developers)
Until native Substrate lifecycle hooks are introduced, actor application developers should adopt these mitigations:
- Use
FullSnapshot Mode for In-Memory OTel Workloads:snapshotsConfig: onPause: Full onCommit: Full
- Configure a Shorter Push Export Interval:
Reducesdkmetric.WithInterval()from the default 60s down to 5s or 10s to ensure metrics and spans are exported promptly during short active execution windows before idle suspension. - Application
SIGTERMSignal Handling & Flush Pattern (Issue Handle Pod Eviction/Deletion gracefully #23):
Register a POSIXSIGTERM/SIGINTsignal listener in the application entrypoint. Upon receiving termination signals, invokemeterProvider.ForceFlush(ctx)andtracerProvider.ForceFlush(ctx)before process termination. - Application Self-Suspend Flush Pattern:
When an actor requests self-suspension, executemeterProvider.ForceFlush(ctx)immediately before invoking the suspend API call. - Persist High-Value Events to
DurableDirVolumes:
InDatamode, append critical events to aDurableDirfile on disk, replaying and flushing them upon container startup post-resume.
5. Proposed Long-Term Substrate Solutions (Complementary Lifecycle Contracts)
To achieve 100% telemetry continuity across all actor lifecycle events, Agent Substrate requires both of the following complementary system contracts working in tandem:
Part 1: Suspend/Resume Hooks (PreSuspend & PostResume — Issue #450)
Handles the pause / checkpoint / resume lifecycle:
PreSuspendHook: Substrate sends an HTTP POST request (/pre-pause) or POSIX signal (SIGUSR1) to the actor container prior to process checkpoint or container pause inDatamode. The actor handler invokesmeterProvider.ForceFlush(ctx)/tracerProvider.ForceFlush(ctx)before returning 200 OK.PostResumeHook: Substrate notifies the actor container post-resume, enabling immediate network connection re-verification or flush triggers.
Part 2: Graceful Termination Contract (SIGTERM Grace Period — Issue #23)
Handles the deletion / teardown / eviction lifecycle:
- Graceful Termination Contract: In
ateom-gvisor/atelet, guarantee a configurable termination grace period (terminationGracePeriodSeconds: 30s) with explicitSIGTERMnotification prior toSIGKILL. This guarantees application OTel SDK signal handlers can execute final flushes to the OTel Collector before container destruction when actors are deleted or garbage-collected.
6. Proposed Action Items
- Align on
PreSuspendhook design in Issue Actor lifecycle hooks #450 and graceful termination contract in Issue Handle Pod Eviction/Deletion gracefully #23. - Update Substrate documentation guide in
docs/observability.mdadvisingonPause: FullandSIGTERMflush handlers for push-exporter workloads. - Provide a reference demo for actor telemetry continuity.
Activity
sanskar-singh-2403 commentedon Jul 24, 2026
hey Zoe Zhao (@zoez7) Da Huang (@git286) Davanum Srinivas (@dims) can i pick this up?
git286 commentedon Jul 24, 2026
Hi Sanskar Singh Bhardwaj (@sanskar-singh-2403), thank you so much for volunteering! We really appreciate the help.
We already have a PR in progress for updating the docs/observability.md. For the alignment with the lifecycle hooks, we recently discussed this with Scott Funkenhauser (@sfunkenhauser) and it seems no one is currently working on issue #450, which is a dependency for this work.
Would you be interested in either contributing to issue #450, or working on the reference demo for actor telemetry continuity? Let me know if either of those sounds good to you, and we'd be happy to help you get started!
docs(observability): add actor application telemetry best practices
sanskar-singh-2403 commentedon Jul 24, 2026
Thanks Da Huang (@git286)! The reference demo for actor telemetry continuity sounds like a great fit - I'd like to take that one as #450 seems to be api-machinery and godfathers are under design discussions rn 😄
A couple of scoping questions before I start:
examples/(or a specific subdir), and is there an existing demo I should mirror for structure/conventions?ForceFlushon the OTelMeterProvider/TracerProviderbefore idle suspension), or should I keep it hook-agnostic so it slots into Actor lifecycle hooks #450'sPreSuspend/PostResumeonce those land?Happy to open a draft PR early so I can get direction as I go.
sanskar-singh-2403 commentedon Jul 27, 2026
Da Huang (@git286) requesting your review on this, Thanks!
git286 commentedon Jul 27, 2026
Caveat up front: I'm fairly new to this project myself, so treat these as
suggestions rather than settled convention — worth a second opinion from
Zoe Zhao (@zoez7) and others before you lock in a structure.
1. Location.
demos/, notexamples/— there's noexamples/at the reporoot. I'd add a new directory (e.g.
demos/agent-telemetry) rather thanextending an existing demo: no demo emits OTel today, so adding an OTLP
exporter to
agent-secretwould make a reachable collector a prerequisite fora demo that otherwise doesn't need one.
2. Approach — both. Lead with the current mitigation since it's the only
thing that works today, but keep the flush behind a small seam so #450 becomes
a call-site move rather than a rewrite: a
flushTelemetry(ctx)helper calledfrom the self-suspend path now, and from a
PreSuspendhandler once thatlands.
One thing worth checking first: on Kind the collector is at
opentelemetry-collector.otel-system.svc.cluster.local:4317, but I haven'tverified that an actor sandbox can reach an arbitrary in-cluster Service —
today only
ateapiandateletare pointed at it. Quick spike before youbuild much, since the answer shapes the rest.
Draft PR early sounds great.
HavenXia commentedon Jul 27, 2026
FYI
agent-secretdemo is being removed in #538 due to the authn added to ate-apiserver.sanskar-singh-2403 commentedon Aug 5, 2026
Da Huang (@git286) Thanks for the steer, both notes made sense. Draft is up: PR #750.
Structure follows your suggestions: it lives in its own
demos/actor-telemetry/(not extending an existing demo), and the flush is isolated behind a single
flushTelemetry(ctx)seam called from the self-suspend path today, so #450becomes a call-site move into a
PreSuspendhandler rather than a rewrite.On the reachability spike you flagged: I hit the gap you predicted. The actor
container had no OTLP endpoint, so the SDK fell back to
localhost:4317insideits own sandbox netns and flushed into the void. I set
OTEL_EXPORTER_OTLP_ENDPOINT=http://opentelemetry-collector.otel-system.svc:4317on the ActorTemplate, matching what the ate-system components use on Kind.
[IF REACHABLE:]
With that set, an actor CAN reach the in-cluster collector: after driving a few
requests and waiting for the idle flush,
ate-demo-actor-telemetrytelemetryshows up in the collector debug exporter and in Jaeger/Prometheus. So arbitrary
in-cluster Services are reachable from the sandbox today, at least for egress to
a ClusterIP with cluster DNS.
[IF NOT REACHABLE:]
Even with the endpoint set, the actor still cannot reach the collector: no
ate-demo-actor-telemetrytelemetry lands, and from inside the actor[DNS resolution / TCP dial to :4317] fails. So the sandbox cannot reach
arbitrary in-cluster Services yet, which matches your read that only
ateapi/atelet/atenet-router are wired up. Flagging this as a dependency; happy
to help scope what egress the actor netns needs.
Opening as a draft so you can direct from here.
demos: add actor telemetry continuity reference demo (agent-substrate…
demos: add actor telemetry continuity reference demo (agent-substrate…