Repository navigation
perf(build): fw-lib (framework library) object compiles still miss zccache cross-project #986
Description
Context
Follow-up to #966 (PR #985). #985 made ESP32 sketch + core compiles hit zccache across project directories (warm compile-sketch 94.5 s → 0.73 s, core 12.9 s → 0.55 s) by fixing two fbuild-side determinism leaks: the object filename hashed the absolute source path, and library include order came from unsorted read_dir.
Framework-lib (fw-libs) object compiles still miss cross-project — warm fw-libs stays ~150 s. The 126 fw-libs TUs report verdict=Cold in the per-compile zccache DIAG (their context key is not found), while the 126 sketch/core TUs now verdict=Hit.
What's ruled out
- Args: a fw-lib compile's full argv (
Esp.cpp, 435 tokens) is byte-identical between the/tmp/ndsand/tmp/nds2checkouts after normalizing the project-dir prefix — including-o, all-I, defines, and order. So it is not an arg-level path/order leak. - Object filename: fw-lib sources live in the global framework cache (
~/.fbuild/…), so fix(build): ESP32 sketch+core compiles hit zccache cross-project (#966) #985'sobject_hash_keykeeps their absolute (already project-independent) path — object names match across checkouts. - worktree_root: fix(build): ESP32 sketch+core compiles hit zccache cross-project (#966) #985 also wires
ZCCACHE_WORKTREE_ROOTon thezccache wrapCLI path fw-libs use; it did not change the outcome.
Hypothesis
Since the args are identical yet the context key differs (verdict=Cold), the leak is a non-arg context input specific to the zccache wrap CLI path (fw-libs use the CLI wrapper → daemon IPC, not the in-process embedded service that sketch/core use). Candidates: per-session system_includes discovery differing on the CLI path, or a session/key_root resolution difference between the CLI-wrap session and the embedded ephemeral session.
Next diagnostic
Dump the fw-lib compile's context-key inputs (not just the hash) for cold vs warm — reuse the [FB966DIAG] tracing approach (patch a tracing::info! at pipeline/mod.rs verdict, [patch] fbuild's zccache to the local clone). Diff the normalized source/includes/defines/system-includes fed to compute_context_key between the two checkouts; the one differing input is the leak. Then relativize/normalize it (fbuild-side) or exclude it (zccache-side), matching #985's approach.
Acceptance
- A clean cold→warm harness run shows warm
fw-libshitting zccache (drops from ~150 s to seconds), completing cross-project caching for all firmware compile phases.
Activity
zackees commentedon Jul 6, 2026
Corrected root cause: fw-libs bypass zccache entirely (not a key leak)
Re-diagnosed. fw-lib (framework builtin library) compiles are not cached by zccache at all — they run gcc directly.
crates/fbuild-build/src/esp32/orchestrator/build.rs:46:let compiler_cache: Option<PathBuf> = None;(set to None by the perf(zccache) Phase 4 stage 2: embedded mandatory — delete FBUILD_ZCCACHE_EMBEDDED + wrapper fallback #800 embedded-service migration).Noneflows tocompile_framework_builtin_libs→library_compiler::compile_library_with_jobs→compile_one_source, which callswrap_compiler_args(&args, compiler_cache). Withcompiler_cache = None,wrap_compiler_argsreturns the args unwrapped, andrun_commandspawns gcc directly — nozccache wrap, no embedded service.Esp.cppI diffed is a core file compiled via the embedded service, which is why it hit.)This is a #800 migration gap: #800 moved every compile to the in-process embedded
ZccacheServiceand deleted thezccache wrapbinary path — but library compilation infbuild-packages::library::library_compilerwas never migrated; it still shells out viarun_command.Fix options
fbuild-packagescan't depend onfbuild-build(dependency direction), so thread a compile-dispatch trait object / closure fromfbuild-build(which owns the embeddedcompile_backend) intocompile_library_with_jobs, so lib TUs dispatch through the same embedded service as sketch/core. Then fix(build): ESP32 sketch+core compiles hit zccache cross-project (#966) #985's object-name + worktree_root fixes make them hit cross-project automatically.zccache wrapfor libraries: pass a resolved zccache CLI path ascompiler_cache. Simpler, but routes lib compiles to the standalone zccache daemon (a different cache store than the embedded service) and contradicts perf(zccache) Phase 4 stage 2: embedded mandatory — delete FBUILD_ZCCACHE_EMBEDDED + wrapper fallback #800's embedded-only intent.Scope: this is an architectural change (compile-path migration), not a determinism tweak like #985. Its payoff is the ~150 s one-time-per-checkout fw-lib cost; the interactive sketch/core win already shipped in #985.
EOF
fix(build): route framework-lib compiles through embedded zccache (#986…