Repository navigation
perf(zccache): firmware compiles don't hit zccache cross-project even when args are workspace-relative #966
Description
Context
Follow-up to #952 / PR #965. After making ESP32 fw-libs compiles workspace-relative (source/-o/-I relativized, cwd = workspace — the same treatment sketch/core already get), the #942 harness shows only a modest warm improvement (fw-libs 185s → 141s), not the expected drop to seconds.
Measuring a clean cold→warm in one container run (fresh ~/.fbuild before cold, fresh project copy for warm), none of the firmware phases hit the cache cross-project:
- warm
compile-sketch≈ 80 s (not a hit) - warm
compile-core-variant≈ 10 s (not a hit) - warm
fw-libs≈ 141 s (partial at best)
So the premise "sketch/core hit cross-project, make fw-libs like them" is only half true: sketch/core are workspace-relative but still don't reliably hit zccache across two project directories in a clean run. Workspace-relative args are necessary but not sufficient.
Investigate
- Does the embedded zccache key include the absolute compile CWD (or another project-specific input) in addition to the relativized args? If so, cross-project hits are impossible regardless of arg relativization.
- Does the embedded zccache persist its object store across builds within a container run at all, and where (
~/.fbuild/<mode>/zccache/vs the soldr-managed store)? - Instrument the per-TU zccache lookup key for one sketch source across
/tmp/nds(cold) and/tmp/nds2(warm) and diff the exact differing component.
Acceptance criteria
- A clean cold→warm run shows warm
compile-sketch,compile-core-variant, andfw-libshitting zccache (sub-second / near-copy). - Root cause (the project-specific key input) documented.
- Warm total drops toward
config-parse + linkonce the compile phases hit.
Decisions
- Priority: P2 — this is the actual lever for the meta: profiling-driven build performance burndown (Docker Linux, NightDriverStrip, cold vs hot cache) #942 warm-scenario win; PR perf(build): compile ESP32 framework libs workspace-relative (cross-project cache stability, #952) #965 is the necessary precondition (workspace-relative args) but the payoff is gated here.
Activity
zackees commentedon Jul 5, 2026
Empirical update — env-var approach does NOT work for the embedded service
Tried setting
ZCCACHE_WORKTREE_ROOT=<compile_cwd>in the per-compile env (both the embeddedsvc.compilepath for sketch/core and thezccache wrapCLI path for fw-libs). Measured in the harness: no change — a clean cold→warm still shows fw-libs≈161s, compile-sketch≈95s, compile-core-variant≈11s (no cross-project hits). Reverted.Root cause of the non-effect: the embedded
ZccacheServicecomputes its cache key from its own context/config, not from the per-compile client env. The client env is forwarded to the spawned compiler subprocess, but does not reachresolve_worktree_root/ key-root computation. So an env var on the compile request is inert for keying. (The earlier analysis conflated the CLI-wrap path, which reads process env, with the in-process service path.)What the fix actually needs
The embedded
ZccacheService::compileAPI must accept a per-requestworktree_root(orkey_root) so fbuild can pass each build's workspace explicitly — a daemon-level env var can't work because one long-lived daemon serves many project dirs. That is a zccache-crate change (add the field to the compile request + thread it into key computation), then fbuild passescompile_cwdthrough it.Until then, PR #965's workspace-relative arg alignment is the extent of the fbuild-side improvement (warm fw-libs 185s→~141s). The sub-second warm win is gated on the zccache API change.
Decisions
zccachecrate (zackees/zccache) first, then a small fbuild follow-up to pass the per-request worktree root.zackees commentedon Jul 6, 2026
PR #985 makes ESP32 sketch + core compiles hit zccache cross-project. Root cause was two fbuild-side determinism leaks embedding the absolute project path into each TU's context key: (1) the object filename's hash suffix was derived from the absolute source path (so the compiler's
-oarg differed per checkout), and (2) library include order came from unsortedread_dir. Both fixed to be project-workspace-relative / sorted.Measured (clean cold /tmp/nds → warm fresh /tmp/nds2): warm compile-sketch 94.5 s → 0.73 s, compile-core-variant 12.9 s → 0.55 s — DIAG confirms identical context key + verdict=Hit across checkouts.
Framework-lib (
fw-libs) compiles still miss cross-project via a separate non-arg input on thezccache wrapCLI path (args byte-identical yet context key differs) — split to #986. #966 stays open until #986 lands the fw-libs hit.fix(build): make ESP32 sketch+core compiles hit zccache cross-project (…
fix(build): make ESP32 sketch+core compiles hit zccache cross-project (…
fix(build): route framework-lib compiles through embedded zccache (#986…
zackees commentedon Jul 6, 2026
Fully resolved. #985 fixed sketch+core cross-project caching (object-filename + library-order determinism); #987 (#986) routed fw-libs through the embedded zccache service. All ESP32 firmware compile phases now hit zccache cross-project: warm sketch 0.8 s, core 0.5 s, fw-libs 2.2 s (from 94.5 s / 12.9 s / 152 s). No zccache-side change was needed — the mechanism worked; the leaks were all fbuild-side.