Repository navigation
perf(build): framework libs (~150s) recompiled per project; core cache never hydrates #952
Description
Context
#942 Phase-1 baseline, warm scenario (global cache fully warm, fresh project copy): fw-libs=150.7s re-runs in full for every new project directory even though the framework libraries are identical for a given platform + flag set. The fw-libs fingerprint only short-circuits within the same project (hot = 1 ms).
Related: core-cache-hydrate=0 ms in all 9 baseline runs while compile-core-variant always spends ~11 s and core-cache-store runs afterwards every time — i.e. the global core cache is written but apparently never hit, even cold → warm → hot inside one container session.
Proposal
- Scope the framework-lib artifact cache globally (keyed by platform/toolchain/flag fingerprint) so a fresh project hydrates ~150 s of compiled framework objects, mirroring the package disk cache design.
- Diagnose and fix the core-variant cache: why store-then-never-hydrate? (key mismatch across builds is the likely culprit.)
Acceptance criteria
- Warm scenario (fresh project, warm cache)
fw-libsdrops from ~150 s to ≤ 5 s in the meta: profiling-driven build performance burndown (Docker Linux, NightDriverStrip, cold vs hot cache) #942 harness. -
core-cache-hydrateactually hits on the second build with identical config (andcompile-core-variant≈ 0 s). - Warm total drops from ~435 s toward ~compile-sketch + link.
- Before/after harness numbers in the PR description.
Part of #942.
Activity
zackees commentedon Jul 5, 2026
Root cause (diagnosed via the #942 harness)
Measured on NightDriverStrip esp32 demo, WARM scenario (fresh project dir + fully warm zccache), post-#951/#955 main:
So with a warm zccache, sketch and core already collapse to sub-second; the entire warm-scenario cost is fw-libs not hitting zccache. The core-cache hydrate (
core-cache-hydrate=0ms) is a secondary issue — irrelevant once zccache is warm, since core-variant hits zccache anyway.Why fw-libs misses zccache while sketch/core hit
The two compile paths construct the invocation differently:
crate::compiler::compile_source(crates/fbuild-build/src/compiler.rs:658-711): computescompile_cwd = zccache::compile_cwd_from_output(output)(the workspace root), relativizes source /-o/-Ito it (path_arg_for_compile_cwd,normalize_flags_for_compile_cwd), and runs withcwd = workspace. → zccache key is workspace-relative → identical for/tmp/ndsand/tmp/nds2→ HIT.library_compiler::compile_one_source(crates/fbuild-packages/src/library/library_compiler.rs:305-397): passes absolutesource(:343),-o(:345), and absolute-I(:531) strings, wraps with the legacyzccache wrapbinary, and invokesrun_command(&args, None, …)—cwd = None(:377). It never callscompile_cwd_from_output/path_arg_for_compile_cwd/normalize_flags_for_compile_cwd. → the zccache key embeds/tmp/nds/...→ MISS on every fresh project.Fix
Make fw-lib compiles workspace-relative like sketch/core. Two options:
crate::compiler::compile_source(the embeddedZccacheService, standardized in perf(zccache) Phase 4 stage 2: embedded mandatory — delete FBUILD_ZCCACHE_EMBEDDED + wrapper fallback #800) instead ofcompile_library_with_jobs, then archive the resulting objects. Stays infbuild-build(ESP32-only blast radius), reuses the proven relativized path.compile_cwd_from_output,path_arg_for_compile_cwd,normalize_flags_for_compile_cwd) fromfbuild-build/src/zccache.rstofbuild-core::path(both crates depend on it), then relativize incompile_one_sourceand passSome(cwd)torun_command. Fixes it for all platforms' library compiles, but wider blast radius (shared path — warrants full cross-platform CI + Windows path-normalization review, since the helpers carry the ESP32 compile on Windows: 'Cannot create temporary file in C:\Windows\' — uncovered tempfile callsite #875/fix(compile): pin TMP/TEMP for compiler subprocess on Windows (#875) #885/fix(compile): force/in path args so cc1 spec-file pass works on Windows #890/fix(compile): rewrite backslashes on the PIO no-compile-CWD fallback path #912 slash fixes).Either way fw-libs drops from ~164 s to seconds with no new archive cache. I attempted a self-contained global archive cache twice; both failed (the cross-project key still differed, and the store didn't write) — the archive cache is the wrong layer. The path-normalization lesson is captured in
agents/docs/path-conventions.md(PR #961).Secondary follow-up
core_cache_keystill computes a project-varying key even after stripping project-Idirs (an additional project-specific input remains — needs a debug-logged component diff to pinpoint). Low priority: core-variant hits zccache when warm, so the hydrate only matters on a cold-zccache fresh machine, and even then the hydrated.cmdhash(written with the storing project's includes) mismatches the hydrating project's per-TU check. Track separately from the fw-libs fix above.feat(dylint): ban raw Path::{starts_with, strip_prefix} in production…
zackees commentedon Jul 5, 2026
PR #965 lands the necessary first step — fw-libs compiles are now workspace-relative like sketch/core (via opt-in
compile_cwdin the shared library compiler; every other caller unchanged). Measured warm fw-libs 185s→141s in the #942 harness. It is NOT the full win: a clean cold→warm shows sketch/core don't fully hit zccache cross-project either, so workspace-relative args are necessary but not sufficient. The remaining payoff is tracked in #966 (firmware-compile zccache cross-project key stability). Keeping #952 open until #966 lands the sub-second warm fw-libs.zackees commentedon Jul 7, 2026
Closing this tracker as split/resolved.
The warm fresh-project
fw-libstarget is resolved by #987/#986: the #942 harness result in #987 shows warmfw-libsdropping from 152s to 2.2s, with sketch/core/fw-libs all hitting zccache cross-project.The bundled
core-cache-hydrateconcern was not proven fixed; it is now separated into #995 so this broad tracker does not keep mixing the resolved fw-libs bottleneck with a lower-priority cache-path audit.Refs #992.
fix(daemon): satisfy the workspace lint gate in the new profiling module
fix(daemon): satisfy the workspace lint gate in the new profiling module
feat(daemon): adopt mimalloc-pprof and prove heap, on-CPU, and off-CP…
refactor(paths): move debug prefix mapping into its own module