Repository navigation
LocationActivity evicts running location-scoped background shells at the 60-minute idle TTL, silently stranding the sessions waiting on them #52908
Description
Repro (opencode 2.0.18, measured 2026-10-02)
A session created in a directory; POST /api/shell spawns sleep 5400 there.
| Time (UTC) | Event |
|---|---|
| 20:43 | session created; background sleep 5400 spawned through POST /api/shell?location[directory]=… |
| ~21:43 | location services evicted directory=… (INFO) |
| 21:51:33 | GET /api/shell/{id} → ShellNotFoundError, HTTP 404 |
What happened
A background shell spawned through POST /api/shell?location[directory]=… is killed when LocationActivity evicts its idle location, and the eviction is silent but for one INFO line. A session waiting on that shell's completion to wake it is stranded: it looks idle and healthy, but nothing is left to wake it.
Why it bites
Agents hold long waits in these shells — CI watches, pollers, "wake me when this process exits" strokes — and expect the shell's completion to remain observable afterwards. Two properties combine to make that unsafe past an hour: (1) the TTL is hard-coded (timeToLive??"60 minutes") and refreshed only by durable session events, so a running shell cannot keep its own location alive; (2) the sweep also interrupts agent loops still active in the location (s.interrupt(id, {reason:"inactivity", awaitSettlement:true})), so even an in-flight multi-hour foreground operation is at risk. A caller can lose the artifact it is blocked on and gets no signal at all — "silently reaped" is indistinguishable from "still running" until someone notices an empty shell bar.
Related work
fix(core): preserve approvals across location cleanup (#46509) made cleanup skip locations with outstanding references, because a permission wait exceeding the timeout lost its request. A running background shell looks like the same kind of outstanding reference from a caller's point of view.
What would fix it (any one)
- Treat a running background shell as location activity, or as an outstanding reference deferring cleanup (the fix(core): preserve approvals across location cleanup #46509 shape);
- make the idle TTL configurable;
- at minimum, make eviction of a location holding running shells visible — a WARN with the shell ids, and/or a
status: "reaped"record rather than a bare 404 — so callers can tell "reaped" from "never existed".
Our workaround
We hold such waits in a detached process outside the server and push a wake on completion (POST /api/session/{id}/prompt, which admirably admits input to a previously idle session and runs the loop). Unnecessary with (1) or (2).
Activity
github-actions commentedon Oct 3, 2026
This issue does not fully meet our contributing guidelines.
What needs to be fixed:
Please edit this issue to address the above within 72 hours, or it will be automatically closed.
This issue might be a duplicate of existing issues. Please check:
If you believe this was flagged incorrectly, please let a maintainer know.
SamSpiri commentedon Oct 3, 2026
Second independent instance, DB-verified (v2.0.22)
Independent confirmation of the exact failure, with the completion notification proven absent from storage.
Setup: worker session
ses_efeb61f29ffeWnthm3xB8kevapin a location (directory) whose only activity was the dispatched turn.Timeline (local time):
shelltool call,background: true, timeout 0 → jobsh_10280cd3f001TXnljvbScsSUtvidle(outcomesucceeded) — last durable event in the locationlocation services evicted(directory)/tmp/.../real2_summary.txtmtime;teefinalised)Evidence the notification was never emitted:
session_message: nosyntheticrow withshellID=sh_10280cd3f...exists anywhere in the DB. If the watcher had delivered, the row would be present (rows are append-only; a later user prompt cannot consume/remove it).session_inbox/session_pendingfor the session are empty — nothing was left undelivered, so this is not a "synthetic admitted but wake dropped" case..outfile (sh_10280cd3f...out) is gone, consistent withshell.ts:234Effect.onInterrupt(() => shell.remove(info.id))running when the watcher fiber is interrupted at eviction.NRestarts=0, no start/exit in journal) —resumeSuspendedSessionsdid not run.Control: a second
background: truejob in the same session,sh_103051812..., ran ~17 min (< 60) and did deliver:syntheticrow withmetadata {source:"shell", shellID, jobID, state:"completed", exit:0}at 20:29:02. So the path works when the location survives; it only breaks when the job outlives the idle TTL.Multiplier detail: the threshold is idle time since the last durable event in the location, swept at 1-min granularity — not job runtime. A background job that ends before ~60 min of location inactivity is safe; one that outlives it loses its completion notification and strands the waiting session, exactly as described here.
The chain in source for this instance:
notifyWhenDoneisEffect.forkIn(scope, …)into the location/plugin scope (tool/plugin/shell.ts:154-186);location-activity.ts:51,77-81interrupts owners withreason:"inactivity"thenlocations.invalidate(ref), closing that scope and interrupting the watcher beforesessions.synthetic(...)runs.packages/core/test/location-activity.test.ts:176-178already asserts a waiting session is evicted after1 + 62minutes of mocked time.github-actions commentedon Oct 6, 2026
This issue has been automatically closed because it was not updated to meet our contributing guidelines within the 72-hour window.
Feel free to open a new issue that follows our issue templates.