Repository navigation
Laptop (amd): proxy systemd units churn with EADDRINUSE while unmanaged copies serve the ports #916
Description
Observed 2026-10-25 on amd (NixOS, user session) by Muse over SSH.
Symptom
systemctl --user is-active for the five proxy units reports nonsense states: clef/codex/antigravity-ls/design-code stuck at activating, agy-proxy inactive — while all five ports (18742-18746) answer /health OK. Drew reports 'lots of crashes and restarts today', but the machine has NOT rebooted (up 14 days, boot Oct 11). Load average sits at ~2.7.
Evidence
- Journal, Oct 25 16:54:39:
clef-proxy-startcrashed with EADDRINUSE on 127.0.0.1:18742; at the same second systemd logged 'Main process exited, status=1/FAILURE' for antigravity-ls-proxy, codex-proxy, clef-proxy and design-code-proxy. systemctl --user show agy-proxy.service: ActiveState=inactive, SubState=dead, Result=success, NRestarts=0 — the unit is cleanly stopped, yet 18743 serves health. So the processes actually listening are NOT the units' processes.- On Oct 21 the listening proxies were a batch of node MainThread processes unrelated to the units' lifecycle (started ~Oct 20), consistent with the warmer/keepalive stack (cf. the wt-warm-* worktrees) spawning its own copies.
Hypothesis
Two supervisors own the same ports: the home-manager systemd user units AND the warmer scripts. The warm copies bind first; unit instances die with EADDRINUSE and cycle through activating/failed, producing the crash/restart churn and the misleading unit states. (Related earlier signal: sync-models.service was in failed state on Oct 21.)
Investigate
- Map listeners to owners:
ss -tlnpPIDs vs each unit's MainPID for 18742-18746. - Find what starts the unmanaged copies (warmer units/scripts, OpenChamber supervisor?) and when (login, boot, respawn loop?).
- Decide a single owner per proxy (units OR warmers) and remove/align the other; check whether the warmers exist because the units were unreliable (e.g. the agy DISABLED marking) — fix the cause, not just the duplication.
- Confirm agy-proxy's intended state (it was marked DISABLED in config, is currently served by a warm copy, and Drew asked for agy working on Oct 21).
Filed from the laptop remote-access thread; the remote-access tracking goal has the surrounding context.
Acceptance criteria
- Every listener on ports 18742-18746 is mapped to its owning process and supervisor (systemd user unit, warmer script, or other), with the evidence recorded on this issue.
- Exactly one owner per proxy port remains: the duplicate supervisor is removed or aligned, and the systemd user units' reported states match reality (
is-activeagrees with what serves the port). - No EADDRINUSE churn in the user journal for the five proxy units over a full day after the fix.
- agy-proxy's intended state is confirmed and matches what is actually running.
Activity
developing-today commentedon Oct 25, 2026
CODE-153
dezren39 commentedon Oct 25, 2026
Investigation findings (2026-10-25, Muse over SSH) — root cause identified
There are three layers trying to own the proxy ports, not two:
Layer 1 — legacy home-manager units (the churn)
clef-proxy.service,agy-proxy.service,codex-proxy.service,antigravity-ls-proxy.service,design-code-proxy.service— plus a sixth,commandcode-proxy.service— are the old home-manager units. All except agy sit inactivating / auto-restartwith MainPID=0: their nix-store start scripts bind, hit EADDRINUSE, die, and systemd restarts them forever (journal shows the FAILURE burst, e.g. Oct 25 16:54:39, all four at the same second). This loop is the "lots of crashes and restarts today", and part of the constant ~2.7 load. The machine itself never rebooted (up 14 days).Layer 2 — the fleet proxy supervisor (the intended owner, for some proxies)
proxy-supervisor.service+proxy-supervisor.timer("Fleet proxy supervisor (hash-keyed A/B reconcile)", every 2 min) manages proxies as systemd-run transient units namedfps-*. Currently active and actually serving:fps-codex-proxy-18744→ codex-proxy (since Oct 24 11:53)fps-design-code-proxy-18746→ design-code-proxy (since Oct 24 11:57)fps-muse-proxy-18747,fps-commandcode-proxy-18748also run under fps (ports beyond the original five)Layer 3 — detached orphans in login-session scopes (the rest)
No fps units exist for clef, agy, or antigravity-ls. Their serving processes are orphans:
node /home/user/code/bin/*.mjsinsession-1115.scope, started Oct 24 11:42, PPID 1.node bin/antigravity-ls-proxy.mjsinsession-1866.scope, cwd~/code, started today Oct 25 05:23 — i.e., something/someone restarted it from a shell session this morning; it survives only as long as that session scope does.agy-proxy.service being "inactive, Result=success, NRestarts=0" while 18743 serves is explained by layer 3: the unit was never the thing serving (the Oct 21
systemctl startinstance ended; the orphan replaced it).Fix direction
*-proxy.servicehome-manager units (all six) — superseded; their only current effect is EADDRINUSE churn and misleading state. (agy's unit is already marked DISABLED in its description yet still loads — check how the others are wired.)sync-models.service(failed state seen Oct 21) — separate, but same "unit state doesn't match reality" smell.No changes were made during this investigation — read-only. The serving proxies never went down; health checks on 18742–18746 pass throughout.
dezren39 commentedon Oct 25, 2026
Fix applied operationally (2026-10-25, Muse over SSH) — single owner achieved
Root cause confirmed: the laptop's deployed home generation was stale. Its unit files still carried
[Install] WantedBy=default.target(current repo config has no Install section by design), so five legacy units were enabled indefault.target.wantsand crash-looped with EADDRINUSE against whoever held each port. The supervisor's config already listed all seven proxies; clef/agy/antigravity-ls were being held by detached session-scope orphans (started Oct 24 11:42 and Oct 25 05:23, apparently by shell sessions during other laptop work), which the supervisor correctly refuses to kill.Actions taken (no rebuild, no service config changes):
systemctl --user disable --nowon clef/agy/codex/antigravity-ls/design-code/commandcode legacy units — wants symlinks gone (0 remain).proxy-supervisor.servicereconcile.Result: all seven proxies now run as supervisor-owned
fps-*units (fps-clef-18742, fps-agy-18743, fps-codex-18744, fps-antigravity-ls-18745, fps-design-code-18746, fps-muse-18747, fps-commandcode-18748), and all seven /health checks pass.Remaining durability item: the laptop's deployed home generation should be refreshed from the current repo config (which drops the [Install] section entirely) at the next system rebuild — planned alongside the declarative muse-user change, after which ownership will be re-verified before closing this issue.
dezren39 commentedon Oct 26, 2026
Intake review
Still live: verified this run — no PR or commit addresses the dual-supervision of the laptop proxy ports (18742–18746); the symptom was observed today on amd and the unmanaged listeners were still serving at filing time. An
## Acceptance criteriasection has been appended to the body (original text preserved verbatim above it).Related: #894 (claude-code-proxy wiring — one of the five proxies in this family), #875 (the claude-code-proxy bridge, merged), #883 (home-layer eval using a stale
bin/tree — the adjacent stale-supervisor symptom in the same home layer).Notes / risks: priority high — this is daily crash/restart churn on the user's own laptop and the misleading unit states hide real outages. The main risk in fixing is removing the warm copies before knowing why they exist (the body flags the agy DISABLED marking): if the warmers are a workaround for unreliable units, deleting them first re-breaks the proxies.
Suggested approach: follow the body's investigate order — map listeners to PIDs/owners first (
ss -tlnpvs each unit'sMainPID), identify what spawns the unmanaged copies and when, then pick a single owner per port and remove/align the other.🤖 Agent note · filed
dezren39 commentedon Oct 30, 2026
Evidence from the #2533 deploy (2026-10-29 ~22:47 CDT), for this issue's two-supervisors picture:
fps-codex-proxy-18744(was PID 3133249, started 2026-10-28 20:57 CDT);codex-proxy.servicesat inactivating/auto-restartwith NRestarts >1250, failing EADDRINUSE every ~3s, MainPID 0 — it serves nothing.codex-proxy.service(cleanly inactive) before restarting the fps unit, specifically so the churn loop could not steal 18744 in the gap. The fps restart + supervisor adoption worked (new PID 3968988, supervisor state adopted:true with the new file hash).codex-proxy.servicewas back inactivating/auto-restartwithout me starting it — some other actor/watchdog re-started it. It still holds no port and does not affect the fps instance, but the churn is self-restoring, which is worth knowing for whoever fixes this issue properly.dezren39 commentedon Nov 2, 2026
🤖 Agent note · handed off
dezren39 commentedon Nov 2, 2026
Triage verdict (2026-11-02, probed live on amd)
No longer reproduces — all four acceptance criteria verified against current main + deployed state.
Listener map (this run,
ss -tlnpvssystemctl --user show):clef-proxy.serviceMainPID matchcodex-proxy.serviceMainPID matchdesign-code-proxy.serviceMainPID matchmuse-proxy.servicecommandcode-proxy.serviceclaude-code-proxy.serviceantigravity-direct.serviceEvidence per criterion:
app.slice/<name>.service, MainPIDs match. Nofps-*units exist; no supervisor units loaded.is-activeagrees with what serves each port. Deployed generation matches repo HEAD (home/common/default.nix): supervisor retired (Proxy service-model redesign: automatic flake services, retire proxy-supervisor ownership, restore consumer Wants #2104, 2026-10-28), per-proxy units are the sole owners.journalctl --user --since '7 days ago' | grep -ci EADDRINUSE→ 0), exceeding the 1-day criterion. All units NRestarts=0.agy-proxy+antigravity-ls-proxydefinitions removed, single Antigravity door isantigravity-directon :18751 (active, serving). :18743/:18745 correctly unbound on localhost.Fix chain: #2104 (per-proxy units, supervisor retired) + 2026-10-30 antigravity-direct cutover + deployed-generation refresh (verified: deployed units match HEAD config). Drew's Oct-25 durability item (refresh deployed generation) is done.
Residual (non-blocking, documented in-repo):
.opencode/opencode.jsoncstill carriesantigravity-agy(:18743) /antigravity-ls(:18745) provider blocks, but both sit indisabled_providers; header notes the sonnet/opus Claude seats still naming antigravity-ls as known-pending. Closing this issue; any re-point work belongs to its own item.🤖 Agent note · triaged