Per-actor log API: actor output is available only through the shared worker pod log #2043

Summary

There is no API that returns the stdout/stderr of a single actor. Actor output ends up in the worker pod's ateom container log, which interleaves every actor that has run on that worker, whatever its atespace, together with worker-level lines. kubectl ate logs actor streams that pod log and filters it on the client side using labels embedded in each JSON line. An integrator that wants to show an actor's logs to the actor's owner therefore has to read, and correctly filter, a stream that holds other tenants' output.

Impact

• Who hits it: every integrator that shows logs to end users, and every operator debugging a single actor.
• What breaks:
  • Showing logs safely to an actor's owner. In the tested integration, the first implementation read the worker pod log and returned lines belonging to other atespaces. Per-actor log viewing was then turned off, and it stays off.
  • Logs across migrations: after an actor resumes on a different worker, its earlier output is in another pod's log, which may already be gone.
  • Access control: reading the worker pod log needs Kubernetes pods/log access in the worker namespace, which is not scoped to an atespace.
• Severity: High.
• Proposed priority: P1. There is no safe workaround for serving logs to tenants, and logs are a basic debugging need.

Environment

• Substrate d277088b (v1alpha), kubectl-ate at the same commit.
• GKE 1.36 with gVisor workers; several atespaces sharing one WorkerPool.

Steps to reproduce / evidence

1. Shared worker log (verified live). Run actors from two or more atespaces on the same WorkerPool, then tail the ateom container log of one worker pod. In one 2000-line tail taken for a single actor, there were lines from two other atespaces and 584 worker-level lines, and the earliest line was about three hours older than the actor.
2. Integrator exposure (verified live). An implementation that returned the worker pod log for "this actor" returned other atespaces' output to the requester.
3. How kubectl-ate does it (code-read at d277088b). In cmd/kubectl-ate/internal/cmd/actor.go, logs actor looks up the actor's current worker pod (:344), streams that pod's whole log (:355), and keeps lines whose JSON labels (logging.googleapis.com/labels or labels) carry the matching actor name and atespace (logLineFilter at :361, filterAndDisplayLogLine at :559-590). It errors unless the actor is RUNNING (:347-349), so it cannot show output from a suspended or crashed actor.
4. Label provenance (code-read). Attribution is added outside the sandbox. internal/actorlog/logger.go (package doc, lines 15-18) forwards each container's stdout/stderr to the worker pod's stdout "annotated with the ate.* identity labels". foldLabelGroups and sanitizeLabels (:204-245) drop any workload-supplied keys under the reserved ate. prefix, from both label spellings. kubectl-ate also strips ate.* labels on display (actor.go:593-600). So the attribution itself is sound. The gap is that filtering happens on the client, over a shared multi-tenant stream that needs unscoped pods/log access, and that no history survives migration or suspend.

Proposed behaviour / API sketch

• A GetActorLogs RPC (unary with paging) plus a StreamActorLogs streaming/follow variant, keyed by atespace and actor name:

  // Unary, paged.
  rpc GetActorLogs(GetActorLogsRequest) returns (GetActorLogsResponse);
  // Streaming; with follow set, keeps streaming new entries.
  rpc StreamActorLogs(GetActorLogsRequest) returns (stream LogEntry);
  message GetActorLogsRequest {
    ObjectRef actor = 1;           // atespace + actor name
    string container = 2;          // optional
    google.protobuf.Timestamp since = 3;
    int32 tail_lines = 4;
    bool follow = 5;               // StreamActorLogs only
    int32 page_size = 6;           // GetActorLogs only
    string page_token = 7;         // GetActorLogs only
  }
  message GetActorLogsResponse {
    repeated LogEntry entries = 1;
    string next_page_token = 2;
  }
  message LogEntry {
    google.protobuf.Timestamp time = 1;
    string container = 2;
    string stream = 3;             // "stdout" or "stderr"
    string line = 4;
  }

• Filtering happens on the server, using the existing ateom attribution (the reserved ate.* labels).
• Authorization is per atespace, consistent with the API authorization model (see ateapi-authorization-and-router-ingress-auth.md).
• History across migrations and after suspend, at least for a bounded retention window.
• Alternatively, per-actor log files or streams on the worker, with attribution added by the platform, that a log agent can ship.

Workaround

An operator with pods/log access can filter the worker pod log by actor name or uid, or use kubectl ate logs actor for a running actor. There is no workaround we would consider safe for serving logs directly to tenants, so an integrator can instead have the workload write its logs to storage the integrator controls.

Additional detail

Related issues:

• Feature request: exec, attach and TTY API into actors (exec-attach-tty-api.md): the other half of interactive debugging.
• Authorization in ateapi, and router ingress authentication (ateapi-authorization-and-router-ingress-auth.md).
• PID-1 exit is not reported: the actor stays RUNNING and keeps its worker (pid1-exit-not-reported-actor-stays-running.md): without logs from a stopped actor, a silent exit is hard to diagnose.