Repository navigation
Quota gatherer ticks fail at import: lib dir resolves to stale openchamber checkout parked on a queue branch #5721
Description
Quota gatherer: every tick fails at import — lib dir resolves to a stale openchamber checkout parked on a queue branch
Symptom (live 2026-11-04 evening): quota-gatherer.service fails every tick in ~13ms with:
quota-gatherer: tick failed before providers ran: quota module import failed: auth.openCodeLocalCredentialSource is not a function. (In 'auth.openCodeLocalCredentialSource({ dbPath })', 'auth.openCodeLocalCredentialSource' is undefined)
(heartbeat ~/.local/state/pm-aggregator/quota-gatherer-heartbeat.json, tickAt 2026-11-05T00:30:13Z, ok:false. Same failure at 18:18 and 18:23 CST ticks; the gatherer stall first observed 2026-11-04 ~18:30:15Z is consistent with this breakage starting earlier that day.)
Root cause: resolveLibDir() in pm-stack/aggregator/scripts/gather-quota.ts picks the first candidate containing quota/index.js: $OPENCHAMBER_LIB_DIR, else ~/git/devtoday-openchamber/... (doesn't exist), else ~/git/openchamber-fork/packages/web/server/lib. The fork checkout ~/git/openchamber-fork is currently parked on branch queue/8357-224 (an old integration merge, e216864bf) with a dirty tree; its packages/web/server/lib/opencode/auth.js predates openCodeLocalCredentialSource. origin/main of the fork has the function (auth.js:214). The gatherer is therefore coupled to the mutable state of an unrelated work checkout — any queue lane that parks that checkout on an old branch breaks quota gathering fleet-wide, silently (heartbeat only).
Verified workaround (used for the #5704 verification run): a clean worktree of the fork at origin/main + OPENCHAMBER_LIB_DIR=<worktree>/packages/web/server/lib makes the gatherer run green.
Fix options:
- Pin the deployed lib: set
OPENCHAMBER_LIB_DIRin the quota-gatherer unit (Nix module) to a dedicated, main-tracking checkout/worktree of the fork that lanes never work in. - Vendor or bundle the quota module with the aggregator so no external checkout is on the runtime path.
- At minimum: fail loud — a tick that dies at import should surface somewhere other than a heartbeat file (the stall went unnoticed for hours).
Do NOT fix by switching/cleaning ~/git/openchamber-fork in place: it holds another lane's dirty queue work (queue/8357-224).
Found during the 2026-11-04 laptop-session coordinator run (Phase 2 of the #5704 verification — that fix itself is landed via PR #5719; the timer stays red until this issue is resolved).
Activity
developing-today commentedon Nov 5, 2026
CODE-1418
dezren39 commentedon Nov 5, 2026
WS-G fix landed 2026-11-04 ~20:00 CST — the merged WS6 fix (PR #5719) is no longer stranded.
Mechanism (verified): quota-gatherer died at import every tick (
auth.openCodeLocalCredentialSource is not a function) because gather-quota.tsresolveLibDir()fell through to~/git/openchamber-fork— a lane checkout parked onqueue/8357-224with a dirty tree, whose builtlib/quota/providers/claude/auth.jspredates the export. On origin/main the export exists atpackages/web/server/lib/opencode/auth.js:214. Its first candidate,~/git/devtoday-openchamber, did not exist.Fix (no disturbance to the parked tree — same branch, same 3 modified files before and after):
~/git/devtoday-openchamber(origin developing-today-forks/openchamber, branch main, clean tree; cloned from local objects, origin URL set to GitHub).~/.config/systemd/user/quota-gatherer.service.d/override.confsetsOPENCHAMBER_LIB_DIR=/home/user/git/devtoday-openchamber/packages/web/server/lib(candidate test #1 in resolveLibDir, ahead of any parked worktree).pm-git-sync-openchamber.timer(5 min) reuses the fleet's ownpm-git-sync.shwithPM_GIT_SYNC_REPOpointed at the new checkout — same discipline as the code primary (ff-only, fails loudly on dirty/off-main, never stashes/discards). First sync pulled it cf089e285a → 5f5d82bd9 cleanly.Verification:
systemctl --user start quota-gatherer.service→ Result=success, ExecMainStatus=0.quota-current.jsonregenerated 2026-11-05T02:00:20Z, heartbeat ok:true, tick 7.2 s. Command Code entry: state ok with PLAN windows measured this tick (monthly 6.64/10, 5h 0/3, weekly 11.07/6 resetting 2026-11-10). Meta entry: retained across the tick's no-data result withretainedFromFailedTick: 2026-11-05T02:00:20.426Zstamped — the #1268/#5719 retention engaging as designed. Caveat: the retained Meta percentages are the post-roll nulls from the 18:30 entry; the Nov 3 100% values were overwritten before the fix landed (they survive in the snapshot history).Follow-ups (not done here): harden
resolveLibDir()to reject dirty/off-main candidate checkouts and log the resolved lib dir + checkout rev at gatherer startup; declare the sync + drop-in in the NixOS/services layer (PR #5764) instead of interim user units.dezren39 commentedon Nov 5, 2026
Regression + durable repair (2026-11-04 ~20:27–20:52 CST, provider audit)
The fix recorded above regressed within the hour, and the mechanism matters:
~/git/devtoday-openchamberwas cloned and ff-pulled to cleanmain(68c9f11c5); ticks verified good (02:00Z, 02:21Z).queue/8357-224back out in that checkout (git reflog:checkout: moving from main to queue/8357-224), returning it to the Oct-21 integration merge (e216864bf) whose quota module lacksfetchQuotaForProviderInstancesand the meta provider.ok:false, service failed).quota-current.jsonfroze at the 02:21Z tick. Same stranding class as the original defect: a shared checkout lanes treat as a working tree cannot be a service's library source.Repair (this time the source is a checkout nothing else uses):
~/git/openchamber-quota-lib(fork origin, branchmain, clean, at 68c9f11c5 at repair time). Verified before pinning:quota/providers/meta.jspresent,fetchQuotaForProviderInstancesexported,opencode/auth.jsexportsopenCodeLocalCredentialSource.quota-gatherer.service.d/override.conf:OPENCHAMBER_LIB_DIR=/home/user/git/openchamber-quota-lib/packages/web/server/lib(backup of the previous drop-in kept alongside).pm-git-sync-openchamber.servicerepointed to the new clone (PM_GIT_SYNC_REPO+ fresh state dir~/.local/state/pm-git-sync-quota-lib); sync run verifiedResult=success, tree clean on main.~/git/devtoday-openchamberwas left exactly as the lane parked it (queue/8357-224) — it is a working checkout again, just no longer a service dependency.ok:true(8,744 ms), all 12 provider runs reporting (details in the provider registry's 2026-11-04 audit section).Follow-up hardening already named on this issue stands (resolveLibDir should reject dirty/off-main candidates and log the resolved rev at startup); add: the gatherer should also refuse a lib whose
quota/index.jslacks the exports it needs, with the missing export named in the failure — an 80 ms import death with an empty perProvider map is how this hid.dezren39 commentedon Nov 5, 2026
🤖 Agent note · done
resolveLibDir()in the gatherer.Triage findings (2026-11-05):
OPENCHAMBER_LIB_DIRis pinned to the dedicated clone~/git/openchamber-quota-lib(nothing else works in it), the sync timer was repointed to it, and the verification tick at 2026-11-05T02:52:34Z reported heartbeatok:truewith all 12 provider runs reporting. No further operational action is owed.gh apiread):resolveLibDir()inpm-stack/aggregator/scripts/gather-quota.ts(line 122) still returns the first candidate containingquota/index.js— no dirty/off-main rejection, no required-export validation, no logging of the resolved dir/rev. That is exactly the stranding class that regressed twice on 2026-11-04 (second time at 20:27 CST when a lane checkedqueue/8357-224back out in the shared checkout).developing-today commentedon Nov 5, 2026
Task 10485 (lane code-s) claimed and assigned to the opencode execution lane. Handed off to the opencode claimant — the bound session drives this task to its terminal state. Parked by this lane per the handoff protocol.
95 remaining items
dezren39 commentedon Nov 9, 2026
🤖 Agent note · parked
dezren39 commentedon Nov 9, 2026
Steward run 252 (2026-11-09T00:20Z / 2026-11-08 18:20 CST): quota-gatherer failing again — NEW failure mode.
opencode quota subcommand failed: ENOENT: no such file or directory, posix_spawn '/home/user/git/opencode-builds/result-ae632cd/bin/opencode'. The result-ae632cd directory exists but bin/opencode is absent.systemctl --user start quota-gatherer.service, 00:20Z) failed identically — heartbeat ok:false, 44 ms, same ENOENT.Cannot find package 'yaml'in the opendesign-renewal leg (non-fatal).dezren39 commentedon Nov 9, 2026
Steward run 253 (2026-11-09T01:22Z / 2026-11-08 19:22 CST): quota-gatherer still down; root cause sharpened.
ENOENT … posix_spawn '/home/user/git/opencode-builds/result-ae632cd/bin/opencode'); a steward restart at 01:22:05Z failed identically (heartbeat ok:false, 16 ms).~/git/opencode-builds/result-ae632cdis a symlink to/nix/store/3fgwv9rr399zjmwd9kqkn2y4wki2rg99-opencode-2.0.30+ae632cd, and that store path no longer exists — the symlink is dangling, sobin/opencodecannot resolve. The build result was presumably garbage-collected or the store was pruned; the gatherer's pinned build pointer needs a rebuild/repoint, not a restart.Cannot find package 'yaml'opendesign-renewal leg error persists in the journal (non-fatal).developing-today commentedon Nov 9, 2026
🤖 Agent note · handed off
dezren39 commentedon Nov 9, 2026
🤖 Agent note · handed off
opencodedezren39 commentedon Nov 9, 2026
🤖 Agent note · triaged
8ac61616-3815-4e7b-b33d-840a3829154a(repo-worker-code) — OpenChamber scheduled laneFresh state (2026-11-09T02:15Z): issue OPEN, no hard-stop labels. Still red, but the described root cause is gone and the live failure is different:
~/git/openchamber-forkis now onmain(clean), andpackages/web/server/lib/opencode/auth.jscontainsopenCodeLocalCredentialSource(2 hits).resolveLibDir()inpm-stack/aggregator/scripts/gather-quota.ts:135is now documented as scoped to the OpenDesign keeper's credential store only — no quota reading resolves from it. Theauth.openCodeLocalCredentialSource is not a functionerror no longer occurs.~/.local/state/pm-aggregator/quota-gatherer-heartbeat.json, tickAt2026-11-09T02:15:03.647Z,ok:false:~/.config/systemd/user/quota-gatherer.service.d/override.confpinsQUOTA_SUBCOMMAND=/home/user/git/opencode-builds/result-ae632cd/bin/opencode quota --json, and thatsymlink is dangling — its Nix store path
3fgwv9rr399zjmwd9kqkn2y4wki2rg99-opencode-2.0.30+ae632cdwas garbage-collected.
result-1add729still resolves to a live store path, so a good build exists to repoint to.Routing: still live and still silent (heartbeat-file only). Fix remains option 3 from the issue plus a pin that cannot dangle:
repoint
QUOTA_SUBCOMMANDto a main-tracking, non-lane checkout/build (or acommand -v opencode-style resolution with anexplicit failure), and surface an import/exec failure on the dashboard rather than only in the heartbeat file. Do not clean or
switch
~/git/openchamber-forkin place.dezren39 commentedon Nov 9, 2026
Root cause found and fixed — two layers, 2026-11-08 evening.
Layer 1 — the gatherer (this issue). The user-unit drop-in
~/.config/systemd/user/quota-gatherer.service.d/override.confpinnedQUOTA_SUBCOMMANDto/home/user/git/opencode-builds/result-ae632cd/bin/opencode— a lane build-result symlink whose Nix store target was garbage-collected (today's full store GC), leaving the symlink dangling. Every tick from 17:45 CST died atposix_spawnENOENT before any provider ran; last good tick 23:45Z. Restarts failed identically, as the steward recorded. Fix: the pin now reads/run/current-system/sw/bin/opencode quota --json— the generation-tracked system-profile binary, which every NixOS generation switch repoints atomically and GC can never dangle (old drop-in backed up alongside). Verified with two consecutive fresh ticks after the fix: 21:53 CST (manual start — heartbeatok:true, 13.0s, quota-current.json regenerated) and 22:00 CST (the timer's own fire — heartbeatok:true, generatedAt 2026-11-09T04:00:18Z, file mtime 22:00:28 CST). Theopencode-go.developing-today-0row is fresh in the record (measured 22:00 CST: 5h 100% · weekly 73% · monthly 40%).Layer 2 — why the Space missed the account and showed stale data. The AI Plans & Usage Space is a static snapshot page; its last build was 01:51 CST, before the account existed (added 16:48 CST) and before PR #7835 registered it in the aggregator's attribution (merged 16:43 CST). No automated refresh job feeds the Space, so nothing picked the account up — and the frozen quota file misled downstream readers for hours (the fleet minder kept reporting quota walls after the 18:00 resets because its source was the pre-reset snapshot). Fix: the Space was rebuilt tonight — OpenCode Go shows ×3 accounts everywhere ($30/mo; monthly headline $336; all-in $383.99 with recomputed shares), the
opencode-go-planscomposite lists all three instances 1:1:1, the Go card carries a per-account current-window status block from the 22:00 CST read, and the per-plan section now prints the measured time with a standing warning that figures older than ~30 minutes mean the gatherer is down.Continuation (Drew's order, 2026-11-08 21:55 CST) — two dedicated OpenChamber tasks now stand guard so this does not silently recur:
7142d25c-0eba-4670-8ce9-4043f02dacb9, every 20 min, America/Chicago, workhorse): checks the record's age each run; past 30 min it diagnoses the gatherer and applies the fix path above, commenting episode evidence here. It runs until the gatherer has survived the next generation switch/deploy AND 24 hours of on-time ticks; only then may it propose its own retirement.2778b851-63d3-46b8-9ee8-81c41d1f63b0, hourly, workhorse): verifies the quota record is <30 min old and every measured provider — developing-today-0 in particular — is registered in the aggregator attribution the Space is built from; repairs registration gaps via the normal PR path or files the blocker with evidence. Runs until 24 consecutive hours of fresh + complete checks pass.Closing: root-cause fix verified on two consecutive ticks and the Space refresh verified with the account rendering. The two guardian tasks carry the watch from here. (Noted separately, not part of this defect: at the 21:53 tick the
_clauderow reported its known expired session, and theopenai.mainrow reported "Session expired — please re-authenticate" — both explicit per-provider error states, surfaced rather than silently stale.)dezren39 commentedon Nov 9, 2026
🤖 Agent note · handed off
dezren39 commentedon Nov 9, 2026
Correction to the task 14282 handoff note (comment 6732809448): fresh state read in the same pass shows issue #5721 is CLOSED (labels include status:done). Per runbook §2 the triage remainder is moot — the claimant (or the next worker on task 14282) should complete task 14282 as already_done with this evidence instead of re-triaging, unless fresh evidence shows the closure was wrong. Lane code-p parked task 14282 for the claimant before this correction was posted.