fbuild daemon dies mid-build with 'error decoding response body' under multi-repo / multi-worktree concurrent use #627

Symptom

fbuild build -e <env> intermittently exits with:

error: daemon error: lost connection to daemon mid-build (error decoding response body);
the daemon process may have died — check ~/.fbuild/prod/daemon/daemon.log

Sometimes seen as a downstream zccache error too:

zccache[err][R]: lost connection to daemon (no response). The daemon may have crashed
or been killed mid-request — run `zccache status` to check, or `zccache stop` then
retry to restart it.

Re-running the same build immediately afterwards typically succeeds. So this is a transient connection or daemon-lifecycle race, not a build-system bug per se.

Reproduction (intermittent, no deterministic repro)

Cumulative across a long session today:

• Working from C:\Users\niteris\dev\fastled3 on branch feat/lpc-drop-float-variant-singleprec-softfp.
• Repeatedly running PLATFORMIO_SRC_DIR=examples/AutoResearch fbuild build -e lpc845brk while iterating on a flash-budget reduction PR.
• Several builds completed cleanly. One failed with the lost connection to daemon mid-build error. A retry of the same command succeeded.

I did NOT manually kill the daemon, stop zccache, or run fbuild daemon stop before any of these failures.

What the daemon log shows

Excerpt from ~/.fbuild/prod/daemon/daemon.log (UTC):

2026-06-18T19:37:42.256Z fbuild_daemon: fbuild daemon starting on port 8765
2026-06-18T19:37:42.258Z fbuild_daemon: found stale daemon PID file pointing at dead PID 36084; cleaning up
2026-06-18T19:37:42.261Z fbuild_daemon: listening on 0.0.0.0:8765
2026-06-18T19:37:42.362Z fbuild_daemon::handlers::operations::build:
    waiting for project lock on .
                              ^^^
                              This is the fastled3 repo (current dir at time of build)
2026-06-18T19:37:42.363Z fbuild_daemon::handlers::operations::build:
    project lock acquired on . after 0 ms
...
2026-06-18T19:37:55.986Z fbuild_build::pipeline::link:
    size: text=57804 data=372 bss=10880 | flash=58176/65536 (88.8%) ram=11252/16384 (68.7%)
2026-06-18T19:37:55.986Z fbuild_build::pipeline::link: build completed in 13.6s
2026-06-18T19:37:56.410Z fbuild_daemon: Daemon is idle; self-eviction in 30s unless new work arrives
2026-06-18T19:38:06.917Z fbuild_daemon::handlers::health: shutdown requested
2026-06-18T19:38:06.917Z fbuild_daemon: shutdown requested via HTTP
2026-06-18T19:38:06.917Z fbuild_daemon: graceful shutdown initiated
2026-06-18T19:38:06.918Z fbuild_daemon: daemon exiting
2026-06-18T19:38:07.526Z fbuild_daemon: fbuild daemon starting on port 8765
2026-06-18T19:38:07.529Z fbuild_daemon: listening on 0.0.0.0:8765
2026-06-18T19:38:07.727Z fbuild_daemon::handlers::operations::build:
    waiting for project lock on C:\Users\niteris\dev\fastled8\.claude\worktrees\feat-fix-esp32c2-rmt5-header\.build\pio\esp32c2
                              ^^^
                              Completely different repo / different worktree

Hypothesis

Two independent issues:

1. Cross-process / cross-worktree daemon contention

The fbuild daemon listens on 0.0.0.0:8765 and serves any client that connects. I had two separate FastLED checkouts active on the same machine:

• C:\Users\niteris\dev\fastled3 (this session's main work)
• C:\Users\niteris\dev\fastled8\.claude\worktrees\feat-fix-esp32c2-rmt5-header (a separate clud worktree)

Both repos / worktrees were issuing builds against the SAME daemon process. The log line at 19:38:07.727 confirms the second repo started a build immediately after the first daemon restarted.

If repo A's build is in flight and repo B sends a shutdown requested to the daemon (via the fbuild daemon stop CLI or a CLI process exit that fires a cleanup hook), repo A's in-flight HTTP connection is severed mid-response. That's exactly the error decoding response body symptom.

2. Idle self-eviction races with stop hooks

The daemon prints Daemon is idle; self-eviction in 30s unless new work arrives at 19:37:56. Eleven seconds later — well inside that 30s window — shutdown requested via HTTP arrives. So either:

• Some CLI client explicitly invoked the shutdown endpoint (looks like our clud / claude-code Stop hooks or a bash test cleanup step might do this), OR
• A race between the idle-self-eviction timer and a final response made the daemon initiate shutdown while there was still a pending build response to send.

The "shutdown requested via HTTP" line specifically suggests an external request, not the self-eviction timer. So the trigger is some other tool in the pipeline.

What would help diagnose / fix

1. Log the HTTP shutdown caller. When /shutdown (or whichever endpoint triggers fbuild_daemon::handlers::health::shutdown requested) is hit, log the requester's PID / cmdline / source IP. Today the line just says "shutdown requested" with no provenance.
2. Refuse shutdown if a build is in-flight. If project lock acquired is held by any client, defer or reject the shutdown until the lock releases. Today the shutdown handler appears to skip this check.
3. Per-project lock / per-port instead of shared :8765. Or include the current working directory hash in the daemon URL so two unrelated repos can't accidentally share a daemon. Today the daemon endpoint: http://127.0.0.1:8765 line in the CLI's stderr output is identical across every project.
4. Client-side retry on error decoding response body. This specific failure mode is recoverable; rerunning the build always works. Auto-retry once before bubbling up.
5. A small status command that shows in-flight clients. Today fbuild daemon ... doesn't have an obvious "who's connected" subcommand; adding one would make field debugging much easier.

Workaround

zccache stop && zccache start followed by retrying the build always recovered the session for me. So did simply rerunning the build a second time.

Environment

• Windows 10 Pro 10.0.19045
• fbuild 2.2.25 (fbuild executable: C:\tools\python13\Lib\site-packages\ci\bin\fbuild.exe)
• fbuild-daemon listening on 0.0.0.0:8765
• Two FastLED checkouts active concurrently on the same machine, only one of which was actively being worked on in any given moment but both could issue builds via tooling hooks.

Related

• Probably interacts with bash bloat / fbuild symbols: no arm-none-eabi path; can't measure LPC845 bloat from FastLED's wrapper #594 (the tooling-gap meta) only in that bash bloat / fbuild symbols: no arm-none-eabi path; can't measure LPC845 bloat from FastLED's wrapper #594 calls out cache + daemon plumbing as the iteration-velocity bottleneck. This issue is independent — it's about daemon resilience under multi-repo usage.