Orphan sweep deletes folded history of sessions owned by other opencode instances (shared store) #782

Short description

Shared store + multiple opencode instances: the orphan sweep checks liveness only against the calling process's own opencode.db, so sandbox-container sessions are treated as deleted and their compartments/historian state are wiped

What happened?

Setup

One Linux host runs multiple OpenCode instances: one directly on the host, plus several
inside docker sandbox containers. All instances share a single magic-context store
(the storage dir is bind-mounted into the containers — this is the documented way to give
sandboxes persistent magic-context state). Each instance has its own opencode.db
(host XDG dir vs. per-container volumes), so no instance can see the others' sessions.

What happened (all times UTC, evidence below)

Session S (ses_ee50efa…, ~2,500 messages, lives only in sandbox A's opencode.db):

• Oct 23 14:19 — historian produces its first compartments for S (after an unrelated
  output-cap fix; runs recorded status=success in historian_runs).
• Oct 23 20:08 — sandbox A's opencode server stops (container idle overnight).
• Oct 24 ~07:20 — a host opencode instance runs the orphan sweep.
  All session-scoped rows for S are deleted: session_meta, compartments,
  historian_runs, message_history_fts, … Store drops from 1 run / 1 compartment to 0/0.
• Oct 24 09:02–09:47 — /ctx-recomp rebuilds everything: 4 chunks, 40 compartments
  (ordinals 1–2386), all status=success.
• Oct 24 17:20:38 — the sweep runs again and deletes all 40 compartments and the 4
  recomp run rows, 7.5 h after they were created.
• Oct 24 17:23 — user messages the session. With folds gone, raw history (~3.4M tokens)
  exceeds the admission limit; no LLM call is even attempted. The session is effectively
  bricked, and the user had to abandon it.

Root cause

sweepOrphanedOpenCodeMessageIndexesInBackground decides liveness by opening the calling
process's own opencode.db (V2StoreReader on the local path) and treating
"not in my db" as "session deleted". In a shared-store multi-instance setup that premise is
wrong: a sandbox session is permanently absent from the host's opencode.db. Protection
then rests entirely on the 24 h freshness guard over message_history_index.updated_at
(MESSAGE_HISTORY_ORPHAN_SAFETY_AGE_MS) — and in our incident that guard did not apply at
all, because the session had no message_history_index rows (the NOT EXISTS (... updated_at > cutoff) eligibility check is vacuously true with zero rows). Two
aggravators:

1. /ctx-recomp and /ctx-wrapup write compartments without refreshing
   message_history_index rows, so fold work never counts as liveness — recomp output is
   exactly as exposed as the folds it replaced (observed: 40 recomp compartments deleted
   7.5 h after creation).
2. There is no config to disable or scope the sweep, so a shared store cannot be defended.

Net effect: in a shared store, any session that is absent from the sweeping instance's
opencode.db and has no fresh message_history_index row loses all folded history to
the next sweep pass — silently (the sweep logs nothing to magic-context.log, and
historian_runs/compartments rows are gone, so /ctx-status shows nothing either).
In our incident the freshness guard did not apply: the session had no
message_history_index rows (vacuously eligible — deleted ~16 h after last chat activity,
while recomp-created compartments were deleted 7.5 h after creation). Host-local sessions
with 12-day-stale index rows survive every sweep, confirming that "missing from my db",
not staleness, is the operative condition.

Store evidence

• message_history_orphan_sweep: harness=opencode2, last_swept_at matches each deletion
  window (Oct 24 ~07:20 and 17:20:38; earlier strike Oct 23 overnight).
• Deletion scope matches SESSION_SCOPED_TABLES exactly (verified against the plugin dist:
  session_meta, compartments, historian_runs, compartment_history_versions,
  session_projects, message_history_fts, pending_ops, …).
• The eligibility query is the only reader of message_history_index.updated_at
  (verified: all other reads of that table select ordinals/dirty-floor, not updated_at).
• Sweep constants in dist: MESSAGE_HISTORY_ORPHAN_SWEEP_BATCH_SIZE=200,
  MESSAGE_HISTORY_ORPHAN_SAFETY_AGE_MS=24h, MESSAGE_HISTORY_ORPHAN_SWEEP_COOLDOWN_MS=10m.

Suggested directions

1. Store-level liveness registry: have each writer record an instance identity
   (pid + opencode.db path) in the shared store, and have the sweep skip sessions whose rows
   were written by another registered/live instance — i.e. make liveness match store scope,
   not process scope.
2. At minimum, a config flag to disable the orphan sweep for shared-store deployments.
3. Separately worth fixing: refresh message_history_index.updated_at when recomp/wrapup
   write compartments, so fold work itself counts as activity. (This alone would NOT have
   prevented the reported loss — the cross-process blindness is the core bug — but it closes
   the staleness half.)

Diagnostics

Manual diagnostics (doctor not run — headless host; and doctor's collection does not
cover cross-process store state).

Plugin: @cortexkit/opencode-magic-context 0.49.20 (opencode2 harness)
Store: single shared context.db (storage dir bind-mounted from host into N docker
containers); WAL mode.

Timeline correlation (UTC):
  Oct 23 20:08  sandbox opencode server stops (last log lines; MCP shutdown cascade)
  Oct 24 07:20  message_history_orphan_sweep.last_swept_at ← first deletion window
                historian_runs: 1→0, compartments: 1→0
  Oct 24 09:02–09:47  /ctx-recomp: historian_runs ids 171–174 all status=success,
                compartments 40 (ordinals 1–2386)
  Oct 24 17:20:38  message_history_orphan_sweep.last_swept_at ← second deletion
                historian_runs: 4→0, compartments: 40→0
                (a new incremental run at 17:27 then reappeared and succeeded —
                 the historian itself was healthy throughout)
  Oct 24 17:23  user message → raw ~3.4M-token history vs 200k admission cap,
                no provider call attempted, session abandoned

Liveness-signal state at time of loss: the affected session had **0 rows** in
message_history_index (post-deletion inspection; consistent with vacuous eligibility and
with strike 1 firing only ~16 h after last chat activity). Control group: host-local
sessions with index rows 12+ days stale survive every sweep (they exist in the host's
opencode.db).

Key code path (dist, v0.49.20): sweepOrphanedOpenCodeMessageIndexesInBackground in the
chunk exporting sweepOrphanedOpenCodeMessageIndexesCooperatively2 —
candidates = session-scoped rows with no message_history_index row newer than 24h;
missing = candidate session_id not in V2StoreReader(own opencode.db).sessionIds();
deletion via deleteSessionScopedRowsSlice over SESSION_SCOPED_TABLES.

Plugin version

0.49.20

OpenCode version

2.1.4

Platform

linux x64

Client

OpenCode TUI (CLI)

Log output (optional)

# magic-context.log contains only "@cortexkit/opencode-magic-context v2 TUI setup" lines
# (container /tmp is wiped on recreate), and the sweep emits nothing to the log on deletion.
# The evidence above was taken from the shared store DB
# (message_history_orphan_sweep / historian_runs / compartments) instead.