perf(zccache): migrate from managed wrapper binary to embedded ZccacheService in fbuild-daemon #789

Goal

Replace fbuild's external managed-zccache binary (zccache wrap … + zccache start + zccache fp …) with an in-process ZccacheService running inside
fbuild-daemon, mirroring the embedded-backend model that soldr landed in
forge-ci/soldr#977.

Current state

• crates/fbuild-build/src/managed_zccache.rs pins MANAGED_ZCCACHE_VERSION
  = 1.12.9 and downloads three binaries (zccache, zccache-daemon,
  zccache-fp) into ~/.fbuild/<mode>/bin/zccache-<ver>/.
• crates/fbuild-build/src/zccache.rs resolves that binary (FBUILD_NO_ZCCACHE
  → FBUILD_ZCCACHE_BIN → managed → env PATH), then per compile:
  • prepends [\"zccache\", \"wrap\", …] via wrap_args,
  • launches a separate zccache start daemon (ensure_running,
    intentionally detached per Daemon teardown hardening: process containment + socket lingering + console events #32 so it outlives the fbuild daemon),
  • shells out to zccache fp check / zccache fp mark-success for
    workspace fingerprints.
• Roughly half of zccache.rs (~300 lines) exists to keep wrapper-process
  cache keys stable: workspace-relative cwd derivation
  (compile_cwd_from_output), --sysroot / -I rewriting
  (normalize_flags_for_compile_cwd), Windows UNC \?\ stripping. All of
  it is only necessary because the cache lives behind a child process.

Why migrate (mirroring soldr's reasoning)

1. One tokio runtime. Embedded service shares fbuild-daemon's runtime,
   so a single tokio-console attach spans build orchestration and the cache
   layer. Today they are two independent processes with separate runtimes.
2. No per-compile exec + IPC overhead. compile_many.rs currently spawns
   one wrapper process per TU; embedded dispatch is an in-process function
   call.
3. Shrinks zccache.rs. Path/cwd/UNC normalization is a workaround for
   wrapper-process semantics. Embedded API takes paths directly.
4. No managed-binary download to maintain. Removes the GitHub release
   dependency, the lockstep version bumps in pyproject.toml /
   MANAGED_ZCCACHE_VERSION / the per-platform asset matrix, and the
   ~/.fbuild/<mode>/bin/zccache-<ver>/ install dance.
5. Removes the intentionally-detached daemon lifecycle. No need for
   zccache start to outlive fbuild-daemon — the cache is the daemon.

Soldr proved the model: CompileBackend::{Wrapped, Embedded} on daemon
state, runtime opt-in via SOLDR_ZCCACHE_EMBEDDED=1, wrapper path kept as
fallback. See soldr commit
41d4753 (forge-ci/soldr#977).

Proposed approach (phased)

Phase 1 — Embedded backend prep (opt-in, default off)

• Add optional embedded Cargo feature on fbuild-build pulling in the
  zccache library as a git rev = \"…\" dep (start with a pinned rev; only
  graduate to a _vender/zccache submodule if iteration friction shows up —
  see "vendoring" below).
• Add CompileBackend::{Wrapped, Embedded} enum, held on a daemon state
  struct in fbuild-daemon. fbuild-build accesses it through a handle in
  the same way orchestrators access other daemon-owned state today.
• Runtime opt-in via FBUILD_ZCCACHE_EMBEDDED=1. Wrapper path stays
  default — zero behavioral change without the flag.
• New crates/fbuild-daemon/src/zccache_embedded.rs wraps ZccacheService
  with fbuild's identity defaults + ~/.fbuild/<mode>/cache/ root.

Phase 2 — Route per-compile dispatch through embedded

• compile_many.rs / compile_exec.rs (or equivalent) gain a path that
  calls ZccacheService::compile(...) directly when
  CompileBackend::Embedded is active. Wrapped path untouched.
• Verify cache-key compatibility: identical artifacts and hit rate between
  wrapped and embedded on the same workspace, or document and justify any
  divergence.

Phase 3 — Embedded fingerprint API

• Migrate check_fingerprint / mark_fingerprint_success off the
  zccache fp CLI to the embedded equivalent. If upstream doesn't expose
  it as a library API yet, file a zccache-side issue.

Phase 4 — Retire wrapper

• Once embedded mode is the default for one release cycle without
  regressions, delete managed_zccache.rs, the wrapper code path in
  zccache.rs, and the path-normalization helpers that only existed to
  serve wrapper cache keys.
• Drop the related env vars (FBUILD_ZCCACHE_BIN,
  FBUILD_ZCCACHE_EMBEDDED) once the wrapper path is gone.

Open design questions

• Where in the dep graph does the embedded service live? Per the
  monocrate policy, no new crate. Two viable spots: (a) a module under
  fbuild-daemon (matches soldr — long-lived daemon owns the service),
  or (b) a module under fbuild-build behind the feature flag. Soldr put
  it under soldr-cli/src/daemon/; in fbuild that maps to
  fbuild-daemon/src/zccache_embedded.rs, with fbuild-build getting a
  handle through the daemon state struct it already consumes.
• Vendoring. Soldr learned the hard way (f176c31 → c2014cc →
  2f14453) that flat-vendoring zccache into _vender/zccache/ is wrong;
  the working model is a git submodule for fast iteration or a plain
  git-pinned Cargo dep when no upstream changes are needed. Recommend
  starting at the plain git-pin and only adopting the submodule if a
  cross-repo change forces it.
• Cache-key compatibility. The wrapper-mode helpers
  (compile_cwd_from_output, normalize_flags_for_compile_cwd, UNC
  stripping) normalize keys to be workspace-relative. Embedded mode must
  preserve that normalization (either inside ZccacheService or as a
  preprocessing step in the fbuild call site) or hot caches built with the
  wrapper will miss after switching.
• Concurrency. compile_many.rs parallelizes compiles by spawning N
  wrapper processes today. Embedded must support parallel in-process
  compiles without forcing a single-mutex bottleneck — confirm
  ZccacheService is Sync and the appropriate handle pattern.
• fbuild-cli boundary. The CLI is a thin HTTP client and must not
  depend on fbuild-daemon. The embedded backend lives entirely on the
  daemon side; the CLI sees no API change.

Test plan / acceptance

• Default builds (no flag) continue to use the wrapper; all existing
  tests in zccache.rs still pass.
• New integration test: with FBUILD_ZCCACHE_EMBEDDED=1, build
  tests/platform/uno, verify a second build hits cache, and verify
  via process snapshot that no zccache.exe was spawned.
• Wrapped vs embedded produce identical object files on
  tests/platform/uno + tests/platform/esp32dev (or document the
  diff with rationale).
• Cold + warm timing comparison published in the PR — quantify the
  per-TU wrapper-spawn overhead this removes.
• soldr cargo clippy --workspace --all-targets -- -D warnings clean
  with and without the embedded feature.

References

• soldr#977 — embedded backend prep, the CompileBackend::{Wrapped, Embedded} enum, SOLDR_ZCCACHE_EMBEDDED=1 runtime opt-in
• soldr commits f176c31 / c2014cc / 2f14453 — vendoring strategy
  evolution; flat-vendor is the anti-pattern, submodule + path-dep is the
  destination
• crates/fbuild-build/src/zccache.rs — current wrapper-mode call site
• crates/fbuild-build/src/managed_zccache.rs — current managed-binary
  download
• Daemon teardown hardening: process containment + socket lingering + console events #32 — original rationale for the intentionally-detached
  zccache start (becomes moot under embedded)