Upgrade zccache to latest (with zccache#1710), flush the embedded backend on daemon exit, and wait on a daemon that is still starting #1480

Fixes the root cause of #1462: the client reports daemon did not become healthy after 3 spawn attempts while the embedded zccache backend takes 12–47 s to start. This issue also covers the fbuild-side persistence gap and the client-side wait that #1462 asks for.

Background

• The daemon binds its port early, but it only serves /health after CompileBackend::start().await returns (crates/fbuild-daemon/src/main.rs:194). zccache startup time therefore counts directly against the client's fixed budget of 3 × 10 s (crates/fbuild-cli/src/daemon_client.rs:973).
• Embedded backend bring-up takes 12-47 s with no logging; metadata.bin never persisted (starts empty every time) forge-ci/zccache#1652 was fixed by fix(embedded): timed bring-up, depgraph load after readiness, periodic metadata snapshot (#1652) forge-ci/zccache#1710 (merge commit 806687f6ff7783d789b9bb86823ba8119b07a263). That PR found that the silent 12–47 s was the dependency-graph load inside start. The load now runs after readiness: status and ping answer at once, and compiles wait up to 60 s for the load, then run as cache misses. The PR also logs a duration for each startup phase and saves metadata.bin periodically, after 32 new contexts or on a timer interval.
• fbuild is pinned to zccache 1.13.1 (rev 8cf6dd0ef5d6f2ff2980039290b9e498fb5a6b0e), 300 commits behind the fix. 1.14.14, the latest release, does not contain it: 1.14.14 was published at 06:53 UTC on 2026-09-26 and ci: full / windows / Check hits its 45-min timeout-minutes on routine runs #1710 merged at 11:15 UTC. Pin the ci: full / windows / Check hits its 45-min timeout-minutes on routine runs #1710 merge commit or later, or the first release that includes it (≥ 1.14.15).
• The zccache PR also shows why metadata.bin was missing on every start: "fbuild's daemon exits without flush/shutdown/drop, so each start began empty." The daemon ends with std::process::exit(0) (crates/fbuild-daemon/src/main.rs:579), and the backend sits in the OnceLock in compile_backend::install_global, which is never dropped. With the 30 s idle self-eviction, a daemon that compiles fewer than 32 contexts can still exit before a periodic save runs.

Tasks

1. Upgrade zccache to the latest version

• Repin zccache in every crate that uses it (fbuild-build, fbuild-build-engine, fbuild-build-arm, fbuild-build-esp, fbuild-build-mcu) to the latest zccache that contains ci: full / windows / Check hits its 45-min timeout-minutes on routine runs #1710.
• Move the running-process git rev in the root Cargo.toml to the exact rev pinned in zccache's workspace Cargo.toml. If the revs differ, rust-lld fails with duplicate rp_*_public symbols. See the comment above the dependency.
• Fix any API changes between 1.13.1 and the new version in crates/fbuild-build-engine/src/zccache_embedded.rs / compile_backend.rs.
• Check the cache-root version directory (~/.fbuild/<mode>/zccache/v<version>/). A version bump starts a new, empty cache root, so the first build after the upgrade is cold. Mention this in the release notes.

2. Flush the embedded backend before the daemon exits

• Call flush() (or shutdown) on the global compile backend before every clean exit(0) in the daemon, including the idle self-eviction path, so metadata.bin, the depgraph and the index are saved on every exit.
• Test: start the daemon, compile, and let it exit cleanly. Assert that metadata.bin exists and that the next start logs a non-empty metadata restore.

3. Client waits on a daemon that is still starting (#1462 ask 1)

• Serve a starting health state as soon as the port is bound, before the USB overlay fetch (main.rs:149), the root-ownership lock wait (main.rs:159, up to 10 s) and backend start.
• When the client finds a daemon that owns the port and reports starting, it polls with a longer deadline that tracks startup progress, instead of respawning and counting down 3 × 10 s.

Acceptance

• A cold fbuild deploy with a large existing zccache cache root reaches /health within about 1–2 s of the daemon's listening on log line.
• metadata cache file not found, starting empty appears only on the first start after the cache root is created, not on every start.
• The daemon log shows zccache's per-phase startup timing events.
• Client gives up on a daemon that is still starting: 'did not become healthy after 3 spawn attempts' while embedded zccache takes 40-47 s to become ready #1462 no longer reproduces under the FastLED bash autoresearch workload.

Related

• Client gives up on a daemon that is still starting: 'did not become healthy after 3 spawn attempts' while embedded zccache takes 40-47 s to become ready #1462: client-side symptom (closed by this issue)
• Embedded backend bring-up takes 12-47 s with no logging; metadata.bin never persisted (starts empty every time) forge-ci/zccache#1652, fix(embedded): timed bring-up, depgraph load after readiness, periodic metadata snapshot (#1652) forge-ci/zccache#1710: upstream root cause and fix
• daemon: concurrent fbuild invocations stampede daemon spawn — add single-flight + bind before heavy init #1010: early bind before heavy init