Repository navigation
Queued pending_ops from finished (sub)agent sessions accumulate forever: no reclamation path exists #763
Description
Provenance: base checked against origin/master @ c2edab548 (fetched 2026-10-22). Store measurements come from a long-running install (0.45.0-era base, disclosed as fork-store numbers; the code paths cited are master's). A stock-artifact reproduction was not run before filing; the steps below are provided, and I will run them as a follow-up if useful.
Summary
ctx_reduce stamps a drop by inserting a row into pending_ops; the row is consumed by the owning session's own later passes. A session that stops running, which is the normal end state for every subagent session, never revisits its queue, and no code path ever clears it. The table grows monotonically for the lifetime of the store. Measured on a single machine's store after ~38 days of use: 1,732 queued rows across 110 sessions, of which 1,497 rows sit in 100 subagent (child) sessions that never ran another pass after their last stamp.
The gap is mechanical and verifiable on origin/master: every DELETE FROM pending_ops site in the codebase requires the owning session to run again. There is no dormancy GC and no cleanup on session.deleted. This holds including >=0.46.0, whose #619 fix (subagent execute-pass drain) makes live sessions drain faster but still prices every reclamation on a future pass of the same session.
Mechanism (master @ c2edab5)
Producers: the stamp path queues rows per session; the drop parameter is tag-number based ("Tag IDs to drop: 3-5...", tools/ctx-reduce/tools.ts:43), so the model can stamp during any session, including a subagent run. Subagent stamping is not an oversight but a documented design law of the current product: transform.ts:1175-1176 gates the §N§ prefix + ctx_reduce guidance on tool availability, "NOT on subagent status" (ctx_reduce is registered process-globally and is callable by a subagent unless the parent's explicit tools allow-list filters it out), and the maintainers' own law test asserts it: transform.test.ts:2447, "Unit B: subagent with ctx_reduce enabled DOES get §N§ prefix (self-management)". A default OpenCode task subagent has the tool. So the producer end is working as intended; only the reclamation end is missing.
Every consumer requires the session to run again:
| Site | Fires when |
|---|---|
apply-operations.ts remove-on-apply |
session runs an apply pass |
storage-ops.ts removePendingOp / clearPendingOps |
called from the above, or the compaction-off transition (compaction-off-transition.ts:293) for a session that is running |
compaction.ts:15 session-wide clear |
a compaction event inside a running session |
storage-tags.ts fold retarget deletes |
a Pi fold pass inside a running session |
session.deleted is the only lifecycle event the plugin handles, and its handler clears only an in-memory dreamer cap map (index.ts:875-877); nothing touches pending_ops, tags, or session_meta. Finished subagent sessions are not deleted at all: the host keeps them for listing and explicit resume, so even that event would not fire.
Row validity is not the issue: every one of the 1,732 rows still addresses an active tag (joined on tag number; the pending_ops.tag_id column stores tag numbers, not the tags.id rowid). They are legitimately queued work for sessions that will never come back.
Evidence (live store, queries 2026-10-22)
- 1,732 rows / 110 sessions with rows; store era ~38 days.
- 1,497 rows / 100 sessions are host child sessions (subagents,
session.parent_idnon-null). 235 rows / 10 sessions are top-level. - By MC's own ledger: 1,207 rows in sessions with no
transform_decisionsrows in the retained log; 385 rows in sessions that ran no pass after the row was queued; 140 rows in sessions that did run later passes and self-consume while they keep running. - Daily queue volume during heavy weeks: 55 to 283 rows/day; the table only ever shrinks inside sessions that keep running.
- The subagent contribution is structural, not historical: a stamp resolves against the calling session's own tool-call registry (
tool-call-registry.ts:25-38:WHERE session_id = ? AND message_id = ?, called with the child's session id), and 163 child sessions hold 7,802 call-id-keyedtagsrows such a lookup can hit;queuePendingOphas no session-class gate. Of 120 subagent sessions that ranctx_reduce, 100 left surviving rows (83%, measured on a pre-Subagents never apply queued ctx_reduce drops between the execute threshold and the force band #619-fix base; expect lower on >=0.46.0, not zero, see "What this is NOT"). The fix must be reclamation, not an addressing grammar.
Per-session end validation (all 100 residue sessions, host message ledger + MC decision rows):
| end class | sessions | detail |
|---|---|---|
| finished clean, stamps queued after last in-run bust | 93 | no error on last assistant message; run simply ended |
ContextOverflowError / APIError prepared prompt exceeds Engine max_context |
5 | last stamp 5 s to 1m55s before the kill |
| engine-shutdown / aborted | 2 | stamp 0-1 s before death |
The death class shows the same starvation race at its worst: two overflow-killed sessions carry a final transform_decisions row of execute, materialized=0, emergency=1: the fold was permitted for the next bust, the next request died over the window limit first, and the run never returned. Whatever the drain policy, a session that ends with queued rows (clean or killed) strands them; the end of the run is the leak boundary.
Investigation notes (including false starts, so the numbers can be reproduced)
- A first census claimed 1,626 "orphan" rows pointing at already-dropped tags. That census joined
pending_ops.tag_idagainsttags.id; the column actually stores tag numbers (seeretargetPendingOpsmatchingtag_idagainst tag numbers, and the apply loop indexing tag status maps by tag number withpendingOp.tagId). With the correct join (tags.tag_number = pending_ops.tag_id AND session_id) there are zero non-active targets. Any "already-dropped orphan" number you see for this table likely carries the same join error. - A "self-heal" claim (
already_reducedreaps stale rows on the next apply pass) was checked row-by-row againsttransform_decisions: true for sessions that keep running, unreachable for the ones that stopped; the pile-up is the set of stopped sessions. - Dormancy signal:
session_meta.last_response_timereflects real provider responses (no spurious refresh), so it is safe to GC on; absence of passes intransform_decisionsis the cross-check. - Reproduction trap: hand-written epoch-ms constants for 2026 dates (a 1.76e12 prefix is 2025, not 2026) silently misclassify era partitions. Every time boundary here was derived with
strftime('%s', ...)/datetime(...,'unixepoch'); hardcoded-epoch reproductions can silently disagree.
What this is NOT (adjacent reports, checked against current master)
- Subagents never apply queued ctx_reduce drops between the execute threshold and the force band #619 (closed, "Subagents never apply queued ctx_reduce drops between the execute threshold and the force band"): a live-session drain-timing bug, fixed in 0.46.0 by giving subagent execute passes the reclaim ride. Complementary, and our 83% rate is measured on a pre-fix base, so the residue rate on >=0.46.0 is lower. It does not close this gap, because every post-fix drain still requires a future pass of the same session: (a) stamps queued after the session's last execute pass strand regardless of version, exactly the end-validation table above (last stamp 5 s to 1m55s before an overflow kill; 93 clean-ended sessions carrying tail stamps); (b) a subagent that never crosses the execute threshold never gets a pass to ride; (c) abandoned top-level sessions (235 rows here) never had the ride question answered at all.
- Orphan sweep covers
message_history_indexonly — the other 23 session-scoped tables leak when sessions are deleted out-of-band #346 (closed, orphan-sweep coverage): the backstop sweep fires for sessions deleted out-of-band and, on current master, still reaps onlymessage_history_index(bookkeeping tablemessage_history_orphan_sweep,storage-db.ts:1759; nopending_opsdelete site outside the session-run paths). Even if extended, our class never reaches it: sessions that stop running are not deleted, and child sessions in particular are never deleted (see Settled Magic Context child sessions are not deleted on OpenCode 2 (teardown delete is directory-scoped; orphan sweep reads the v1 session table) #558/EXPERIMENTAL_BACKGROUND_SUBAGENTS: hidden sessions created inside ephemeral subagent processes can never be deleted at runtime (owner is undefined, the DELETE is never even attempted) #594). - Show how much subagent tool-call context ctx_reduce cannot reclaim #643 (open, subagent unreclaimable skeletons): a visibility feature for retained
skeleton_realarguments; its body itself reports subagent stores with "no pending ops left" post-Subagents never apply queued ctx_reduce drops between the execute threshold and the force band #619. Different resource, different fix.
Proposed fix (laws, not a prescribed mechanism)
A queued row is only reachable by its session's future passes. Add a dormancy reclamation: on storage open, clear pending_ops for sessions whose last recorded provider response is older than a TTL (we use 30 days). Sessions resurrecting after a month pay one re-stamp; the scheduler re-derives drop candidates from tag state anyway, so nothing else is lost. The sweep must treat "never responded" (NULL or 0) as not-dormant. We run a working implementation of this law in our fork as a best-effort boot statement next to the existing lease-heal: one prepared DELETE ... WHERE session_id IN (SELECT session_id FROM session_meta WHERE last_response_time > 0 AND last_response_time < cutoff), law-tested in both directions.
Precise prevention for the subagent class, alongside the sweep: discharge the queue at the death moment. OpenCode emits per-session session.status events (idle/busy/retry, packages/schema/src/session-status-event.ts) to plugins; a busy-to-idle transition on a session with isSubagent set is exactly "this run is over and detached" (a child's only future is explicit task_id resume). Clearing that child's pending_ops on idle removes the 83% class at the source, no TTL needed. Clear, not apply: application rewrites earlier wire for a wire that will never be requested again, and any real resume re-derives drop candidates anyway. Scope must stay child-only (top-level sessions go idle after every turn), and because the event is ephemeral, a missed event still needs the dormancy sweep as backstop, which is also the only mechanism covering abandoned top-level sessions. Note subagent_invocations (with ended_at) tracks MC-internal runners, not host task children, so session.status is the host-side end signal to use.
Secondary, smaller and orthogonal: on session.deleted, purge the session's pending_ops (and arguably tags/session_meta) rows; today the handler clears only the dreamer cap map. This does not cover subagents (they are not deleted), which is why the dormancy sweep is the load-bearing fix.
Reproduction (stock, any >=0.45.x)
- Fresh profile, start a session, have the model run
ctx_reducestamping a few tag numbers on large tool outputs (rows land inpending_ops). - Run that work inside a subagent (e.g., the task tool) so the stamping session ends after its run.
- Restart the app (confirms boot does not reclaim) and query the store:
SELECT COUNT(*) FROM pending_ops WHERE session_id = '<child-session-id>'still returns the stamped rows, and the child session shows no further passes. - Optional long-horizon: on a heavy install,
SELECT COUNT(*) FROM pending_opsis monotone over weeks.
Not claimed
- No claim that any running session's legitimate backlog is wrong; the live-sessions portion self-consumes by design.
- No measurement of storage cost: the rows are small; the defect is unbounded growth with zero reclamation, not present-day size.
- Rust transform mode queue behavior: not inspected for this report.
Sources
origin/master@c2edab548(fetched 2026-10-22):tools/ctx-reduce/tools.ts:43,index.ts:875-877,compaction.ts:15,compaction-off-transition.ts:293,storage-ops.ts:42,51,storage-tags.ts:2169,2180,2195,2359,tool-call-registry.ts:25-38,transform.ts:1175-1176,transform.test.ts:2447.- Live store
context.db+ hostopencode.dbjoins (queries 2026-10-22): dormancy census, growth buckets, per-session end validation (message.dataerror JSON), call-id-keyedtagsregistry counts.
Activity
Sugaroverdose commentedon Oct 22, 2026
Post-filing update on the implementation side, found while building the prevention path:
session_meta.last_response_time(Date.now() milliseconds) against a seconds-converted cutoff, so no row ever matched. The unit bug survived our own tests because the fixtures were hand-written in seconds. Fixed: cutoff now stays in the writer's unit and fixtures use real units. The law as stated above (and the census numbers) is unaffected - every time boundary in the measurements was SQL-derived (strftime/datetime), not a hardcoded epoch.session.idledischarge from "Proposed fix" is now implemented in the same fork build: onsession.idle(the v1 event name; the docs' v1.18.30 example is literally "send notification on session completion"), oneSELECT is_subagent FROM session_meta WHERE session_id = ?then, iff it is a child session, delete itspending_ops. Top-level sessions idle after every turn and keep their backlog; the dormancy sweep stays as the backstop for missed ephemeral events and abandoned top-level sessions.magic-alfonso commentedon Oct 22, 2026
Thanks for the careful census, @Sugaroverdose. The numbers and the join correction are right: these rows are real queued drops for sessions that haven't run again. We're closing this as working as designed. Here's why.
Queued drops are session state, like tags and summaries. A
ctx_reducestamp records a decision the agent made about its own context, and it applies on that session's next pass. A finished subagent isn't gone: the host keeps it, and it can be resumed with itstask_id. When it resumes, its queued drops apply on the next rebuild, as the agent intended. Clearing them on idle or after a timeout would silently throw away decisions a resumed session still relies on. The drop candidates the scheduler derives from tag state don't cover explicitctx_reducechoices.When a session really ends, the queue goes with it. On current master,
session.deletedruns the full session cleanup (event-handler.tscallsclearSession), which deletes every session-scoped table. That includespending_ops, which is inSESSION_SCOPED_TABLES, along withtagsandsession_meta. The handler atindex.ts:875-877that the report cites is a second listener for the dreamer cap map, not the only one. OpenCode sessions deleted while the plugin isn't running are picked up by the orphan sweep, which clears the same tables for sessions missing from OpenCode's own session table.Cost: the rows are a few dozen bytes each. 1,732 rows after 38 days is well under a megabyte, and it's bounded by the sessions the host keeps.
If a host session is still listed but will never be resumed, deleting it in the host removes its Magic Context state too.