Skip to content

Queued pending_ops from finished (sub)agent sessions accumulate forever: no reclamation path exists #763

Closed as not planned

Description

@Sugaroverdose

Provenance: base checked against origin/master @ c2edab548 (fetched 2026-10-22). Store measurements come from a long-running install (0.45.0-era base, disclosed as fork-store numbers; the code paths cited are master's). A stock-artifact reproduction was not run before filing; the steps below are provided, and I will run them as a follow-up if useful.

Summary

ctx_reduce stamps a drop by inserting a row into pending_ops; the row is consumed by the owning session's own later passes. A session that stops running, which is the normal end state for every subagent session, never revisits its queue, and no code path ever clears it. The table grows monotonically for the lifetime of the store. Measured on a single machine's store after ~38 days of use: 1,732 queued rows across 110 sessions, of which 1,497 rows sit in 100 subagent (child) sessions that never ran another pass after their last stamp.

The gap is mechanical and verifiable on origin/master: every DELETE FROM pending_ops site in the codebase requires the owning session to run again. There is no dormancy GC and no cleanup on session.deleted. This holds including >=0.46.0, whose #619 fix (subagent execute-pass drain) makes live sessions drain faster but still prices every reclamation on a future pass of the same session.

Mechanism (master @ c2edab5)

Producers: the stamp path queues rows per session; the drop parameter is tag-number based ("Tag IDs to drop: 3-5...", tools/ctx-reduce/tools.ts:43), so the model can stamp during any session, including a subagent run. Subagent stamping is not an oversight but a documented design law of the current product: transform.ts:1175-1176 gates the §N§ prefix + ctx_reduce guidance on tool availability, "NOT on subagent status" (ctx_reduce is registered process-globally and is callable by a subagent unless the parent's explicit tools allow-list filters it out), and the maintainers' own law test asserts it: transform.test.ts:2447, "Unit B: subagent with ctx_reduce enabled DOES get §N§ prefix (self-management)". A default OpenCode task subagent has the tool. So the producer end is working as intended; only the reclamation end is missing.

Every consumer requires the session to run again:

Site Fires when
apply-operations.ts remove-on-apply session runs an apply pass
storage-ops.ts removePendingOp / clearPendingOps called from the above, or the compaction-off transition (compaction-off-transition.ts:293) for a session that is running
compaction.ts:15 session-wide clear a compaction event inside a running session
storage-tags.ts fold retarget deletes a Pi fold pass inside a running session

session.deleted is the only lifecycle event the plugin handles, and its handler clears only an in-memory dreamer cap map (index.ts:875-877); nothing touches pending_ops, tags, or session_meta. Finished subagent sessions are not deleted at all: the host keeps them for listing and explicit resume, so even that event would not fire.

Row validity is not the issue: every one of the 1,732 rows still addresses an active tag (joined on tag number; the pending_ops.tag_id column stores tag numbers, not the tags.id rowid). They are legitimately queued work for sessions that will never come back.

Evidence (live store, queries 2026-10-22)

  • 1,732 rows / 110 sessions with rows; store era ~38 days.
  • 1,497 rows / 100 sessions are host child sessions (subagents, session.parent_id non-null). 235 rows / 10 sessions are top-level.
  • By MC's own ledger: 1,207 rows in sessions with no transform_decisions rows in the retained log; 385 rows in sessions that ran no pass after the row was queued; 140 rows in sessions that did run later passes and self-consume while they keep running.
  • Daily queue volume during heavy weeks: 55 to 283 rows/day; the table only ever shrinks inside sessions that keep running.
  • The subagent contribution is structural, not historical: a stamp resolves against the calling session's own tool-call registry (tool-call-registry.ts:25-38: WHERE session_id = ? AND message_id = ?, called with the child's session id), and 163 child sessions hold 7,802 call-id-keyed tags rows such a lookup can hit; queuePendingOp has no session-class gate. Of 120 subagent sessions that ran ctx_reduce, 100 left surviving rows (83%, measured on a pre-Subagents never apply queued ctx_reduce drops between the execute threshold and the force band #619-fix base; expect lower on >=0.46.0, not zero, see "What this is NOT"). The fix must be reclamation, not an addressing grammar.

Per-session end validation (all 100 residue sessions, host message ledger + MC decision rows):

end class sessions detail
finished clean, stamps queued after last in-run bust 93 no error on last assistant message; run simply ended
ContextOverflowError / APIError prepared prompt exceeds Engine max_context 5 last stamp 5 s to 1m55s before the kill
engine-shutdown / aborted 2 stamp 0-1 s before death

The death class shows the same starvation race at its worst: two overflow-killed sessions carry a final transform_decisions row of execute, materialized=0, emergency=1: the fold was permitted for the next bust, the next request died over the window limit first, and the run never returned. Whatever the drain policy, a session that ends with queued rows (clean or killed) strands them; the end of the run is the leak boundary.

Investigation notes (including false starts, so the numbers can be reproduced)

  1. A first census claimed 1,626 "orphan" rows pointing at already-dropped tags. That census joined pending_ops.tag_id against tags.id; the column actually stores tag numbers (see retargetPendingOps matching tag_id against tag numbers, and the apply loop indexing tag status maps by tag number with pendingOp.tagId). With the correct join (tags.tag_number = pending_ops.tag_id AND session_id) there are zero non-active targets. Any "already-dropped orphan" number you see for this table likely carries the same join error.
  2. A "self-heal" claim (already_reduced reaps stale rows on the next apply pass) was checked row-by-row against transform_decisions: true for sessions that keep running, unreachable for the ones that stopped; the pile-up is the set of stopped sessions.
  3. Dormancy signal: session_meta.last_response_time reflects real provider responses (no spurious refresh), so it is safe to GC on; absence of passes in transform_decisions is the cross-check.
  4. Reproduction trap: hand-written epoch-ms constants for 2026 dates (a 1.76e12 prefix is 2025, not 2026) silently misclassify era partitions. Every time boundary here was derived with strftime('%s', ...) / datetime(...,'unixepoch'); hardcoded-epoch reproductions can silently disagree.

What this is NOT (adjacent reports, checked against current master)

Proposed fix (laws, not a prescribed mechanism)

A queued row is only reachable by its session's future passes. Add a dormancy reclamation: on storage open, clear pending_ops for sessions whose last recorded provider response is older than a TTL (we use 30 days). Sessions resurrecting after a month pay one re-stamp; the scheduler re-derives drop candidates from tag state anyway, so nothing else is lost. The sweep must treat "never responded" (NULL or 0) as not-dormant. We run a working implementation of this law in our fork as a best-effort boot statement next to the existing lease-heal: one prepared DELETE ... WHERE session_id IN (SELECT session_id FROM session_meta WHERE last_response_time > 0 AND last_response_time < cutoff), law-tested in both directions.

Precise prevention for the subagent class, alongside the sweep: discharge the queue at the death moment. OpenCode emits per-session session.status events (idle/busy/retry, packages/schema/src/session-status-event.ts) to plugins; a busy-to-idle transition on a session with isSubagent set is exactly "this run is over and detached" (a child's only future is explicit task_id resume). Clearing that child's pending_ops on idle removes the 83% class at the source, no TTL needed. Clear, not apply: application rewrites earlier wire for a wire that will never be requested again, and any real resume re-derives drop candidates anyway. Scope must stay child-only (top-level sessions go idle after every turn), and because the event is ephemeral, a missed event still needs the dormancy sweep as backstop, which is also the only mechanism covering abandoned top-level sessions. Note subagent_invocations (with ended_at) tracks MC-internal runners, not host task children, so session.status is the host-side end signal to use.

Secondary, smaller and orthogonal: on session.deleted, purge the session's pending_ops (and arguably tags/session_meta) rows; today the handler clears only the dreamer cap map. This does not cover subagents (they are not deleted), which is why the dormancy sweep is the load-bearing fix.

Reproduction (stock, any >=0.45.x)

  1. Fresh profile, start a session, have the model run ctx_reduce stamping a few tag numbers on large tool outputs (rows land in pending_ops).
  2. Run that work inside a subagent (e.g., the task tool) so the stamping session ends after its run.
  3. Restart the app (confirms boot does not reclaim) and query the store: SELECT COUNT(*) FROM pending_ops WHERE session_id = '<child-session-id>' still returns the stamped rows, and the child session shows no further passes.
  4. Optional long-horizon: on a heavy install, SELECT COUNT(*) FROM pending_ops is monotone over weeks.

Not claimed

  • No claim that any running session's legitimate backlog is wrong; the live-sessions portion self-consumes by design.
  • No measurement of storage cost: the rows are small; the defect is unbounded growth with zero reclamation, not present-day size.
  • Rust transform mode queue behavior: not inspected for this report.

Sources

  • origin/master @ c2edab548 (fetched 2026-10-22): tools/ctx-reduce/tools.ts:43, index.ts:875-877, compaction.ts:15, compaction-off-transition.ts:293, storage-ops.ts:42,51, storage-tags.ts:2169,2180,2195,2359, tool-call-registry.ts:25-38, transform.ts:1175-1176, transform.test.ts:2447.
  • Live store context.db + host opencode.db joins (queries 2026-10-22): dormancy census, growth buckets, per-session end validation (message.data error JSON), call-id-keyed tags registry counts.

Activity

  1. Sugaroverdose commented on Oct 22, 2026

    @Sugaroverdose
    Author

    Post-filing update on the implementation side, found while building the prevention path:

    • Our fork's dormancy sweep, as shipped when this issue was filed, was inert: it compared session_meta.last_response_time (Date.now() milliseconds) against a seconds-converted cutoff, so no row ever matched. The unit bug survived our own tests because the fixtures were hand-written in seconds. Fixed: cutoff now stays in the writer's unit and fixtures use real units. The law as stated above (and the census numbers) is unaffected - every time boundary in the measurements was SQL-derived (strftime/datetime), not a hardcoded epoch.
    • The session.idle discharge from "Proposed fix" is now implemented in the same fork build: on session.idle (the v1 event name; the docs' v1.18.30 example is literally "send notification on session completion"), one SELECT is_subagent FROM session_meta WHERE session_id = ? then, iff it is a child session, delete its pending_ops. Top-level sessions idle after every turn and keep their backlog; the dormancy sweep stays as the backstop for missed ephemeral events and abandoned top-level sessions.
  2. magic-alfonso commented on Oct 22, 2026

    @magic-alfonso

    Thanks for the careful census, @Sugaroverdose. The numbers and the join correction are right: these rows are real queued drops for sessions that haven't run again. We're closing this as working as designed. Here's why.

    Queued drops are session state, like tags and summaries. A ctx_reduce stamp records a decision the agent made about its own context, and it applies on that session's next pass. A finished subagent isn't gone: the host keeps it, and it can be resumed with its task_id. When it resumes, its queued drops apply on the next rebuild, as the agent intended. Clearing them on idle or after a timeout would silently throw away decisions a resumed session still relies on. The drop candidates the scheduler derives from tag state don't cover explicit ctx_reduce choices.

    When a session really ends, the queue goes with it. On current master, session.deleted runs the full session cleanup (event-handler.ts calls clearSession), which deletes every session-scoped table. That includes pending_ops, which is in SESSION_SCOPED_TABLES, along with tags and session_meta. The handler at index.ts:875-877 that the report cites is a second listener for the dreamer cap map, not the only one. OpenCode sessions deleted while the plugin isn't running are picked up by the orphan sweep, which clears the same tables for sessions missing from OpenCode's own session table.

    Cost: the rows are a few dozen bytes each. 1,732 rows after 38 days is well under a megabyte, and it's bounded by the sessions the host keeps.

    If a host session is still listed but will never be resumed, deleting it in the host removes its Magic Context state too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions