fix(daemon): embedded zccache holder watchdog aborts active builds at flush and spawn guards #1866

Context

Two consecutive ESP32-S3 AutoResearch builds lost their shared fbuild daemon while compilation was active. The daemon's final diagnostics identify an embedded zccache lock-holder watchdog abort, rather than idle self-eviction, a firmware failure, or evidence that another client killed the daemon.

Environment: Linux, fbuild 2.5.49+git.98affea702ff (98affea702ff58e551124cda68f863659acebb7c), embedded zccache 1.15.8, production daemon endpoint 127.0.0.1:22178. The same failure is visible in both client output and ~/.fbuild/prod/daemon/daemon.log. This interrupted FastLED/FastLED#5001 hardware validation before firmware could be tested.

Failure 1: artifact-index flush guard

Command:

bash compile esp32s3_opi_psram8 --examples AutoResearch --defines FASTLED_OBJECTFLED_DIAGNOSTICS=1

The client compiled framework/core/library files, then failed after 157.2 seconds:

2026-11-02T23:58:18.110637Z ERROR fbuild::output: daemon error: lost connection to daemon mid-build (error decoding response body)
Compilation failed (fbuild) [157.2s]

The daemon had reported Daemon staying alive: build/deploy operation in progress at 23:57:00. Its final diagnostics before the next start were:

zccache: lock hold exceeded its budget. Registered sites:
  Holding Mutex ... age_ns=15025622665 budget_ns=15000000000 at .../zccache-1.15.8/src/artifact/store.rs:492:14 (thread tokio-rt-worker)
  Holding RwLock(read) ... age_ns=2022595990 budget_ns=15000000000 at .../zccache-1.15.8/src/daemon_core/daemon/spawn_exclusion.rs:61:64 (thread running-process-actor)

At 23:59:14 the next ordinary invocation started a daemon and logged found stale daemon PID file pointing at dead PID 3893893. There is no idle-eviction or graceful-shutdown marker between the active build and the watchdog diagnostic.

Failure 2: spawn-admission shared guard

An ordinary canonical AutoResearch retry subsequently failed before flashing, after 133.5 seconds:

2026-11-03T00:01:26.836617Z ERROR fbuild::output: daemon error: request failed: error sending request for url (http://127.0.0.1:22178/api/deploy)
BUILD+FLASH FAIL

This daemon's final diagnostics were:

zccache: lock hold exceeded its budget. Registered sites:
  Holding RwLock(read) ... age_ns=15135884870 budget_ns=15000000000 at .../zccache-1.15.8/src/daemon_core/daemon/spawn_exclusion.rs:61:64 (thread running-process-actor)
  Waiting RwLock(read) ... age_ns=6482680866 budget_ns=15000000000 at .../zccache-1.15.8/src/daemon_core/daemon/spawn_exclusion.rs:61:64 (thread running-process-actor)
  Waiting RwLock(write) ... age_ns=7517021476 budget_ns=15000000000 at .../zccache-1.15.8/src/daemon_core/daemon/server/persist/write_cached.rs:182:34 (thread tokio-rt-worker)

The next normal startup found dead PID 3974966. This is a second terminal abort with a different overdue holder, not a retry of a still-running build. No extra device operations, daemon restart/kill, or cache deletion were performed for this investigation.

Confirmed source and remaining uncertainty

• fbuild embeds zccache in the daemon process: zccache_embedded.rs. The workspace pins zccache = "=1.15.8".
• In zccache tag 1.15.8 (0549fb69a67b53ff2a1c340419bd4cccd0f40815), ArtifactStore::flush holds flush_lock using the short 15-second STATE_TIMEOUT while snapshotting, serializing, writing, file-syncing, renaming, and directory-syncing the index. The diagnostic names this guard.
• spawn_exclusion.rs deliberately gives the shared fork/exec admission guard a 15-second hold budget and the exclusive materialization guard a longer budget. The second diagnostic names the shared guard, with materialization writers waiting behind it.
• sync_watchdog.rs reports the overdue sites and calls std::process::abort(). Because the backend is embedded, this terminates fbuild's shared daemon and every in-flight client connection.

The logs prove which guards exceeded their limits and why the host process terminated. They do not prove whether the underlying delay was slow filesystem I/O, spawn/actor scheduling, a lock-order problem, or another backend defect. Do not attribute this to another agent, OOM, or an idle-timer race without additional evidence. A separate earlier idle-eviction log entry is not the cause of these two events.

Proposal

1. Reproduce the two guard overruns under the real embedded backend, retaining the holder/waiter diagnostics and adding stage timings where the current evidence cannot distinguish I/O from lock ordering or spawn admission.
2. Fix the underlying guard scope/admission problem upstream where appropriate, then pin the proven version in fbuild. Long operation permits and short state critical sections need distinct contracts; do not merely disable the watchdog or raise every lock budget.
3. Ensure a cache/backend failure cannot silently invalidate all independent build/device workflows. Where recovery is possible, report a typed infrastructure failure and preserve unrelated requests; if process isolation is required, justify that boundary separately.

Acceptance criteria

• RED -> GREEN: capture a focused hardware-free failing integration/child-process test using the production embedded service and the implicated flush/spawn paths before implementing the fix; retain the corresponding green result.
• Exercise cold-ish and warm ESP32 builds concurrently in separate project roots, including cached artifact materialization and index flushing, with no daemon abort or lost HTTP build/deploy response.
• Tests distinguish a legitimate long I/O/admission operation from a truly wedged short critical section. Keep watchdog/deadlock diagnostics and bounded failure behavior rather than globally suppressing them.
• Corrupt/truncated cache output must never be accepted as a successful build. Preserve atomic index durability and spawn/materialization exclusion guarantees.
• Re-run the command above and canonical AutoResearch with matching flags through the stock daemon, and record the resolved fbuild/zccache versions and terminal results.

Decisions

• File this separately from fix(daemon): enforce exclusive device leases across deploy and RPC workflows; add bounded sharing queue #1864: lease admission and a process-wide cache watchdog abort are different failure mechanisms.
• This is a confirmed daemon-reliability failure with a known abort site, not a claimed complete diagnosis of the slow holder. Root-cause timing/locking investigation is part of the fix.
• Keep the fix in fbuild/upstream zccache, not in FastLED retry loops, manual daemon restarts, or cache deletion.

Related issues

• fbuild daemon dies mid-build with 'error decoding response body' under multi-repo / multi-worktree concurrent use #627: historical mid-build disconnect symptoms; this report adds concrete lock-watchdog diagnostics on current 2.5.49.
• Upgrade zccache to latest (with zccache#1710), flush the embedded backend on daemon exit, and wait on a daemon that is still starting #1480: the existing embedded zccache lifecycle/version integration.
• zackees/zccache#1962 / #1973: holder-budget watchdog design; these reports expose production overruns at two guarded sites.