Repository navigation
Home-layer eval uses a stale bin/ tree: commandcode-proxy unit 203/EXEC, managed opencode stuck on old rev #883
Description
Found during the 2026-10-25 NixOS switch (generation 504, built from main @45c817e0).
Symptom 1 — declarative commandcode-proxy unit crash-loops (stopped manually after 59 restarts):
commandcode-proxy.service: Unable to locate executable '/nix/store/3gy9alpqrmkvyvjz8x34fwh1knq3dycl-bin/commandcode-proxy-start.sh': No such file or directory
Main process exited, status=203/EXEC
The -bin store copy contains only 16 files — exactly the bin/ tree of the old main (e18acea era). It is missing every bin file added Oct 24–25: commandcode-proxy.mjs/.test.mjs/-start.sh, claude-code-proxy., muse-proxy-start.sh, proxy-relay.mjs, proxy-supervise.. The ExecStart in home/common/default.nix is ${../../bin}/commandcode-proxy-start.sh, and the file IS in the repo (100755, verified via git ls-tree) and in the flake source export (25 files in bin/).
Symptom 2 — OpenChamber's managed opencode is still 2.0.30+e7baa0d after the switch, even after killing it so OpenChamber respawns it (new port, same old store path), while /run/current-system (gen 504) ships 2.0.30+954ff7c.
Deterministic, not a switch-time race: building nixosConfigurations.amd.config.system.build.toplevel from a clean checkout of current main (and from ~/code) reproduces the running toplevel bit-for-bit (2lrvq0pi…), stale -bin included. So somewhere in the eval graph the home layer (home/common and friends) is sourced from a tree that predates the Oct 24–25 bin additions, even though lib.from-root resolves to inputs.self.outPath and the flake source export itself is complete.
Impact: low right now — the proxy-supervisor's transient fps-commandcode-proxy unit serves :18748 (HTTP 200 verified), and the background opencode service (which does serve the new build + the opencode-go-* instances) is unaffected. But the declarative unit cannot take over until this is fixed, and the managed server will not pick up new opencode builds.
Pointers for whoever digs in: check how home-manager.users.user is assembled for amd (nixos/hosts/amd-krdp.nix, nixos/home/default.nix) and whether any input or module list pins a second copy of this repo at an old rev; compare nix flake metadata source bin/ (25 files) against the -bin path baked into the built unit (16 files).
Acceptance criteria
- The root cause of the stale home-layer source tree is identified and named (which input/module pins the pre-Oct-24 tree), not just worked around.
- A clean-checkout build of
nixosConfigurations.amd.config.system.build.toplevelproduces a-binstore copy containing the full currentbin/tree (including commandcode-proxy-start.sh and claude-code-proxy.*). - After a switch, the declarative
commandcode-proxyunit starts without 203/EXEC and serves on its registered port. - OpenChamber's managed opencode picks up the current build after a switch/respawn (version/rev matches the system profile).
Activity
developing-today commentedon Oct 25, 2026
CODE-135
dezren39 commentedon Oct 26, 2026
Intake review
Fleet monitor intake pass (2026-10-25 run). Acceptance criteria appended to the body this pass (original body preserved verbatim above it).
Still live — no resolution found this run: The body establishes this is deterministic (a clean-checkout build reproduces the stale
-bincopy bit-for-bit), so it will not heal on the next switch. No merged PR since filing names the home-layer source pin as fixed.Related: #894 (claude-code-proxy home wiring — its home-manager unit will land in this same home layer; if the layer is sourced from a stale tree, that unit won't take effect either, so this issue may gate part of #894), and the 2026-10-25 switch record (generation 504, main @45c817e0) in the body.
Notes / suggested approach: The body's own pointers are the right start: how
home-manager.users.useris assembled for amd (nixos/hosts/amd-krdp.nix,nixos/home/default.nix), whether any input or module list pins a second copy of this repo at an old rev, andnix flake metadatasource paths vslib.from-rootresolving toinputs.self.outPath. The discriminating experiment: evaluate the home layer'sbinsource path directly (nix evalthe relevant module arg) and compare its rev/file list against the flake source export's 25 files — the stale tree has exactly 16.Risks: Medium investigation risk — eval-graph source pinning can hide in an unexpected input (a locked flake input of a sub-flake is the classic shape). Impact today is contained (transient supervisor unit serves the proxy), but every future home-layer change is silently suspect until this is root-caused, hence priority/high.
dezren39 commentedon Oct 29, 2026
Partial blast-radius fix landed as PR #2092 (open, awaiting CI as of 2026-10-28 evening CDT — checks queued on the backlogged self-hosted runners).
What #2092 changes:
home/openclaw/default.nix— removescommandcode-proxy.service,design-code-proxy.service, andagy-proxy.servicefromopenclaw-gateway'sWants(mcpx.servicekept). The proxies are owned by the proxy-supervisor now (transientfps-*units, canonical ports 18742–18749), so a gateway restart could previously re-pull the manual units — including this issue's crash-loopingcommandcode-proxywith its stale nix-store ExecStart. After this change a gateway restart can no longer resurrect it.Still to do once #2092 merges: deploy via nixos-rebuild switch on amd, restart openclaw-gateway, and verify the manual proxy units stay inactive while the supervisor-owned proxies answer on :18748/:18746.
Not in #2092: the manual unit declarations themselves stay in
home/common/default.nixas documented manual one-off fallbacks. Noted follow-up:openchamber.servicehas the same Wants pattern ondesign-code-proxy.service(plus clef/codex/antigravity-ls manual units) and deserves the same treatment in a separate change.dezren39 commentedon Oct 29, 2026
Follow-up to the manual-proxy-unit
Wantscleanup: PR #2094 — "fix(home): drop openchamber wants on manual proxy units" — is open as the twin of #2092 (which removed the same class ofWantsfromopenclaw-gateway).openchamber.service(home/common/default.nix) declaredWantson four manual per-proxy units; #2094 removes exactly those:clef-proxy.servicecodex-proxy.serviceantigravity-ls-proxy.servicedesign-code-proxy.servicenetwork-online.targetandmcpx.serviceare preserved, theAfterordering entries are untouched, and a comment in #2092's style records why the manual units must not be re-added (proxy-supervisor owns the proxies on canonical ports 18742–18749; OpenChamber reaches them by port; the manual units are stopped fallbacks — the manual design-code unit is the one that fights the supervisor's instance for its taken port). The manual unit declarations themselves are retained as fallbacks, same as in #2092.Status as of 2026-10-28 evening CDT: PR #2094 is open and mergeable, but CI on the self-hosted runners is backlogged —
Nix flake evaluation,run, andtreefmt checkstayed pending through an ~18-minute watch (CodeRabbit skipping), the same queue holding #2092. Per the merge-only-on-green rule it has not been merged, and the deploy step (converge +nixos-rebuild switch --flake …#amd, openchamber restart trigger-verification,systemctl --user reset-failed agy-proxy.service) is deferred until it merges.dezren39 commentedon Nov 2, 2026
🤖 Agent note · triaged — CONFIRMED still actionable (mitigated, not root-caused); fix packet enqueued
$HOME/code/bincheckout ExecStarts — the original 203/EXEC crash-loop is mitigated for the proxies, and the supervisor is retired.lib/default.nix:22-27(root=inputs.self.outPath, self-consistent — the stale tree is NOT via from-root) and no second copy of this repo anywhere inflake.nixinputs, so the stale source is still unexplained.home/common/default.nix:892(sync-models oneshot ExecStart bakes${../../bin/sync-models.mjs}) and:1643+(writeShellScriptBinCLI shims for clef/agy/codex/antigravity-ls/design-code proxies plus sync-models/update-models all bake${../../bin}/*.mjsstore paths). If the home layer ever evaluates from a stale tree again, these go stale the same way.repo-health:v1:fix:developing-today/code:issue:883): run the discriminating eval probe on amd (home-layer bin source file list vs repobin/), name the pin, eliminate or justify the remaining store-path refs. Deploy (nixos-rebuild switch) stays with the operator — explicitly out of scope.27 remaining items
dezren39 commentedon Nov 3, 2026
🤖 Agent note · handed off
dezren39 commentedon Nov 3, 2026
🤖 Agent note · state: handed off
repo-health-code-pilot)ses_e5e5f822132ffegLLPlcfg2x6ropencode-route/workhorse, agentbuildrepo-worker-codedezren39 commentedon Nov 3, 2026
🤖 Agent note · probe results — stale pin named (task 6513, PR #4673)
ses_e5e5f822132ffegLLPlcfg2x6r)inputs.self.outPath(lib/default.nix:22-27), captured at thenixos-rebuildeval that built the booted generation. No second flake input exists:${../../bin/...}in home/common/default.nix andlib.from-root "bin/..."both resolve to${self.outPath}/bin/.... Nix copies that tree into/nix/storeat eval time, freezing the checkout's tree into the generation until the next switch.b0zkpr8...-nixos-system-amd-26.11.20261010runs store snapshotssqhrandkr38r96acv9l0p2qpn80qrdl-sync-models.mjs, while the current checkout hashes to82wlh9r4hz6g7gyfpj0bard1rnymk0k0-sync-models.mjs— DIFFER (live lacks theKNOWN_ARGSblock from PR fix(design-code): repoint :18745 legs to antigravity-direct :18751 (#1413, unblocks opencode#1409) #4626, carries the old sync-models user service fails from Nix store layout: resolves opencode config to /nix/.opencode/opencode.jsonc #591 comment).in7j1gz1...-bin: 36 files incl.codex-proxy.mjs.pre-warm-20261024,*.bak-20261024*,proxy-supervise.mjs.newdebris vs 49 files in the repobin/tree (debris cleaned, antigravity-direct/codedb-bridge/opencode-route-proxy added since).nixosConfigurations.amd … services.sync-models.Service.ExecStartrendersexec /run/current-system/sw/bin/node "$HOME/code/bin/sync-models.mjs"— no store snapshot.grepconfirms zero${../../bin}refs remain inhome//nixos/(only legitimate package${...}/bin/...refs).treefmt --ciclean.basic: OpenCode metered-provider guardis cancelled and its workflow is disabled (gh run rerun→ "Unable to retry disabled workflow"), so green is unreachable without operator action. Deploy/switch stays with the operator per packet.dezren39 commentedon Nov 3, 2026
🤖 Agent note · supplement — shim build verification (task 6513)
ses_e5e5f822132ffegLLPlcfg2x6r)nixosConfigurations.amd): every wrapper execs$HOME/code/bin/<name>.mjs— zero store-bakedbin/snapshots remain inhome.packagesoutput.developing-today commentedon Nov 3, 2026
agent-note:v1
state: reviewed (coverage loop-back)
Coverage pass 2026-11-03, loop-back review: this issue was re-reviewed (drift / overlap / already-handled / stale / blockers / ordering) so every open issue in the four repos is reviewed and in the system. Verdict: still live; in the system as task 6513 (small_fix, parked, priority 70) in repo-health-code-pilot. Labels audited and synced. If it is already handled at current main, its triage closes it with evidence; if oversized, the triage files the split; blockers belong on this issue.
Found via: coverage loop-back 2026-11-03