oc-password-sync fails on every trigger: missing openchamber.service.d/external-opencode.conf drop-in #3724

Activity

1. github-actions

   assigned

   dezren39

   on Nov 1, 2026
2. dezren39

   added

   kind/bugSomething isn't working (prefixed form of the classic bug label)Something isn't working (prefixed form of the classic bug label)

   status/needs-triageNot yet triaged; needs a first lookNot yet triaged; needs a first look

   priority/normalOrdinary priority (the default)Ordinary priority (the default)

   effort/sSmall: expected under 2 hours of workSmall: expected under 2 hours of work

   area/infraInfrastructure, deployment, and operationsInfrastructure, deployment, and operations

   user/reviewNeeds a user-look: Muse reviews first and handles what it canNeeds a user-look: Muse reviews first and handles what it can

   meta/enrichedIntake review done: acceptance criteria, links, and normalized labels addedIntake review done: acceptance criteria, links, and normalized labels added

   on Nov 1, 2026
3. dezren39 commented on Nov 1, 2026

   

   dezren39

   on Nov 1, 2026

   Member

   More actions

   Intake review

   Fleet monitor intake pass (protocol §5), 2026-11-01.

   Related: developing-today-forks/openchamber#894 (the sync's origin, self-assigned there); code#3720 / PR #3722 (flake fold-in — verified it neither causes nor fixes this failure).

   Notes: Filed with no labels and no acceptance criteria. Liveness check: PR #3722 mentions this issue only as context; no merged PR resolves it. The journal evidence (2026-11-01 08:12:26 CST) is precise.

   Risks: The fix location is a genuine decision — installer vs script expectation — so this carries user/review for Muse's review lane.

   Suggested approach: Decide the owning side first (check the #894 setup path for where the drop-in was supposed to be created), then patch that side and re-trigger the .path unit to verify.
4. developing-today commented on Nov 1, 2026

   

   developing-todayBot

   on Nov 1, 2026 – with developing.today

   ContributorAuthor

   More actions

   Resolved 2026-11-01 — root cause was one layer down

   The failing unit was a zombie of a superseded mechanism, not a missing file to restore:

   • The sync was written for openchamber#894: it rewrote the hand-applied drop-in ~/.config/systemd/user/openchamber.service.d/external-opencode.conf and restarted OpenChamber on drift.
   • openchamber#766 then retired that drop-in by design: bin/openchamber-start.sh now reads the password straight from ~/.local/state/opencode/service-prod.json at every OpenChamber start and exports it, so no drop-in should exist. (Timeline matches: last successful sync 2026-10-30 09:18 "in sync"; the drop-in was gone by 2026-10-31 04:20, and every trigger since failed in the rewrite step, which assumed the file exists.)
   • Nobody updated the sync, so it kept firing on every service-prod.json change and exiting 1. Re-creating the drop-in would have resurrected a shadowed artifact — the start wrapper re-reads the JSON and overrides it anyway.

   Fix applied (laptop, ~/.local/bin/oc-password-sync.sh — operator-managed by design; original preserved at ~/.local/state/openchamber/oc-password-sync.sh.orig-2026-11-01)

   The script now judges drift against what OpenChamber actually runs with: the OPENCODE_PASSWORD in the running process's /proc/<MainPID>/environ (falling back to a last-synced hash in ~/.local/state/openchamber/password-sync.state when OpenChamber is down). On drift it restarts openchamber.service — whose wrapper re-reads the JSON — and then verifies the new process holds the new password before exiting 0. The flake units from the #3720 fold (nixos/home/oc-password-sync.nix) are unchanged and remain correct: the .path trigger and oneshot shape are exactly what the fixed script needs.

   Verification (2026-11-01)

   • Unit run: Result=success, log "in sync; no action".
   • .path end-to-end: rewriting service-prod.json with identical content fired the sync (log line at trigger time), exit 0.
   • Drift detection: dry-run against a test JSON with an altered password → "drift detected", exit 0, OpenChamber untouched.
   • Propagation: after a real restart, the new OpenChamber process's OPENCODE_PASSWORD hash matches the current service-prod.json password; OpenChamber active, HTTP 200 on :3000.

   Closing as fixed.
5. developing-today

   closed this as completedon Nov 1, 2026
6. developing-today commented on Nov 2, 2026

   

   developing-todayBot

   on Nov 2, 2026 – with developing.today

   ContributorAuthor

   More actions

   Reconciliation (2026-11-01 22:25 CST) — this closure was correct; the fold notes that seemed to contradict it were stale.

   The fold-completion record (laptop-hygiene Addendum 12) described the live .service as "still fails on trigger for the known pre-existing missing drop-in, code#3724", which read as if this fix had not survived the flake fold. Re-diagnosis against the live system tonight:

   • The active flake unit (FragmentPath → /nix/store/43ccambr…-oc-password-sync.service) runs ExecStart=/home/user/.local/bin/oc-password-sync.sh — the operator script this issue's fix rewrote. The fold replicated that ExecStart exactly; no repo script with the old zombie logic is involved anywhere.
   • Journal, full day: the FileNotFoundError failures on the retired openchamber.service.d/external-opencode.conf occurred at 05:07, 05:11, 05:55 and 08:12 CST — all before the 10:52 rewrite. After it: successes at 10:52:57 and 10:53:05 (the original verification), then post-fold successes at 22:09:28 and 22:16:12 through the store unit.
   • Fresh trigger test tonight (22:25:59 CST): a same-content rewrite of service-prod.json fired the .path; service Result=success, ExecMainStatus=0; OpenChamber MainPID unchanged, active, HTTP 200. A dry-run drift test (temp JSON, OC_DRY_RUN=1) at 22:26 printed drift detected (dry run), exit 0, OpenChamber untouched.

   So the layer fixed here (the operator script) is the layer the live unit executes, before and after the fold. Not reopening.

   One genuine residual did exist, at a different layer: the corrected script had no home in version control — the flake declared a unit whose ExecStart target it did not provide. Fixed in PR #4170: the script is vendored byte-identical beside the module (nixos/home/oc-password-sync.sh, sha256 fc47b0d0ca4766a65a8a9225c865f59a920176a274e259adaa44d2f57d0a5df3 — equal to the live file) and provisioned at the referenced path via home.file. ExecStart and both units are unchanged; the provisioning takes effect at the next activation, and live behavior is already the vendored bytes, so nothing about the running service changes when it lands.