Skip to content

feat(core): Workspace boot janitor — persisted bound workspaces are never idle-suspended or snapshotted after a restart #1622

Closed

Description

@dezren39

Problem

The Workspace service's idle janitor only ever sees workspaces that the current process has connected to. After a restart, persisted workspace rows with live bindings are invisible to it — nothing at boot reconciles them — so their backing resources are never idle-suspended (and the promised ~23h safety snapshot rotation does not exist at all) until someone happens to provision/connect that exact workspace ID again.

The code itself promises this in a later PR that has not landed:

packages/core/src/workspace.ts:358-359 (at main a6bc4dca06faa872d83f1b65b00c4f7cd6dbd6a5):

// TODO(workspace-plan): add the boot janitor and ~23h safety snapshot rotation in a later PR.
// TODO(workspace-plan): make cold wake interruptible with a re-pin loop against janitor races.

Evidence (verified this run, pinned reads at a6bc4dca)

  • The janitor loop (workspace.ts:190-222, repeated on Schedule.spaced(pollInterval ?? 1 minute) at :222) iterates only [...connections.entries()] (:194) — the in-memory connections map, which is empty at process start and only gains entries via open() on provision/connect.
  • Every WorkspaceTable access in the tree is by-ID (find :96, saveBinding :105, updates :184/:211, create select :228-230, insert :244-247, delete :347); there is no scan of persisted rows anywhere, and no boot-time reconciliation pass in the layer.
  • grep -rn "janitor" packages/ --include="*.ts" returns only the two TODO lines; grep -rn "snapshot" packages/core/src/workspace/ packages/core/src/environment/ returns nothing — the safety snapshot rotation is entirely unimplemented, not just unscheduled.
  • packages/core/test/workspace.test.ts (33 tests) pins the in-process lifecycle — including "persists the workspace lifecycle and reconnects after idle suspension" (:330) with a fake driver recording suspendForIdle — but has no boot/restart scenario: no test builds the layer over pre-existing persisted rows (grep for boot/restart/reconcile in the test file: no matches).

Why it matters

Workspaces back real (potentially billed / resource-holding) driver resources. A server restart — deploy, crash, upgrade — silently orphans every persisted, bound workspace that nobody reconnects to: it is never suspended for idleness, never snapshotted, and its row's last_used_at goes stale with no process responsible for it. The longer the deployment runs between restarts, the more this looks like a slow resource leak that only manual destroy calls can clean up. The TODO's second line compounds it: cold wake is not interruptible against janitor races, so a boot janitor added naively could race wakes — the two TODOs need to be designed together.

Acceptance criteria

  • On layer startup (boot), the Workspace service scans persisted WorkspaceTable rows and reconciles them: rows with a binding whose last_used_at is beyond the idle threshold are driven through the same suspendForIdle path the poll loop uses (respecting the per-workspace lock and the deliberate racing-spawn semantics documented at workspace.ts:206); rows with no binding are left untouched.
  • Boot reconciliation is safe against a concurrent provision/connect of the same workspace (no double-suspend, no suspend of an active connection) — the cold-wake re-pin concern in the :359 TODO is addressed or explicitly documented as out of scope with reasoning.
  • The ~23h safety snapshot rotation from the :358 TODO is either implemented (driver capability + rotation cadence + retention) or split into its own follow-up issue with the TODO updated to point at it — the TODO must not silently survive another "later PR".
  • New tests in packages/core/test/workspace.test.ts style (fake driver, persisted rows seeded before layer construction): (a) a stale bound row is suspended at boot without any connect; (b) a fresh bound row is left alone; (c) a boot scan racing a connect does not suspend the active workspace.
  • Existing workspace tests still pass.

Dedup / related

Activity

  1. added
    task/triageQueue packet kind: verify, reproduce, and route this item
    source/creatorFiled by the new-issue-creator job
    priority/normalOrdinary priority (the default)
    on Nov 4, 2026
  2. developing-today commented on Nov 4, 2026

    @developing-today
  3. added
    queue/queuedHas a queued task in the Supabase agent queue
    app/corePackage: core (pkgs/ or apps/ in this repo)
    effort/mMedium: expected under 1 day of work
    on Nov 4, 2026
  4. dezren39 commented on Nov 4, 2026

    @dezren39
    Author

    🤖 Agent note · filed

    • Agent: new-issue-creator job (new-issue-creator-opencode)
    • Session / ID: cron new-issue-creator-opencode, run 2026-11-04T12:35Z
    • Found via: filed this run
    • Task: Filed the Workspace boot-janitor gap: the idle janitor only iterates the in-memory connections map, so persisted bound workspaces are never reconciled after a restart, and the ~23h safety snapshot rotation promised at workspace.ts:358 does not exist.
    • Expectation: effort/m — triage first (boot-reconciliation + cold-wake race design), then implement with seeded-row tests.
    • Links: evidence in issue body (pinned reads at main a6bc4dc); queue packet packet-opencode-1622.json (repo-health-opencode-pilot)
  5. dezren39 commented on Nov 4, 2026

    @dezren39
    Author

    agent-note:v1
    agent: repo-queue-worker-shift-opencode (lane opencode-a)
    task: 9524
    state: done (triaged)
    Triage CONFIRMED on fresh main (workspace evidence unchanged at a6bc4dc): workspace.ts:358 TODO verbatim; the janitor loop (:190-222) iterates only the in-memory connections map (:194, empty at boot) and suspendForIdle (:205) hangs off a live connection; every WorkspaceTable access is by-ID; no boot scan exists; the safety snapshot rotation is entirely unimplemented (no snapshot code in workspace/ or environment/). workspace.test.ts pins the in-process lifecycle but has no boot/restart scenario.
    Follow-up: fix task 9526 enqueued (packet hidden_files/1622-fix-packet-9524.json) covering boot reconciliation through the poll loop's suspend semantics, boot/connect race safety, the (a)(b)(c) test shapes from this issue's acceptance, and splitting the snapshot rotation into its own issue with the TODO repointed.

  6. added
    work/assignedAn agent holds this item via the queue (claimed task)
    and removed
    queue/queuedHas a queued task in the Supabase agent queue
    on Nov 4, 2026
  7. 3 remaining items

  8. added
    queue/parkedQueue task is parked (waiting on CI, time, or an event)
    status/parkedDeliberately paused with a recorded resume condition
    and removed
    queue/claimedA queue worker has claimed the task for this item
    on Nov 4, 2026
  9. dezren39 commented on Nov 4, 2026

    @dezren39
    Author

    Parked by Lane A (task 9526): another session holds the work — its 13:06Z pickup note and implementation commit 37312eabfe on branch queue/9526-1622-workspace-boot-janitor (workspace.ts + workspace.test.ts, pushed 13:12Z) — but no PR is open yet and the queue claim had lapsed when this lane claimed the task. Parked 30 minutes naming that branch as the conflicting artifact; no parallel work was done. Resume brief on task 9526: shepherd the owner's PR if one appears; otherwise salvage the existing branch (test it, open the PR from it) rather than reimplementing.

  10. added
    queue/claimedA queue worker has claimed the task for this item
    queue/parkedQueue task is parked (waiting on CI, time, or an event)
    status/parkedDeliberately paused with a recorded resume condition
    and removed
    status/parkedDeliberately paused with a recorded resume condition
    queue/parkedQueue task is parked (waiting on CI, time, or an event)
    queue/claimedA queue worker has claimed the task for this item
    on Nov 4, 2026
  11. added a commit that references this issue on Nov 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    app/corePackage: core (pkgs/ or apps/ in this repo)area/typescriptTypeScript code areaeffort/mMedium: expected under 1 day of workkind/enhancementNew feature or requestpriority/normalOrdinary priority (the default)queue/parkedQueue task is parked (waiting on CI, time, or an event)source/creatorFiled by the new-issue-creator jobstatus/parkedDeliberately paused with a recorded resume conditiontask/triageQueue packet kind: verify, reproduce, and route this itemwork/assignedAn agent holds this item via the queue (claimed task)

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions