meta: profiling-driven build performance burndown (Docker Linux, NightDriverStrip, cold vs hot cache)

Context

fbuild's end-to-end wall clock on a real project has never been measured systematically under a controlled, reproducible environment. We have anecdotal slow spots (toolchain download/extract, library resolution, sequential install steps) but no profile data separating on-CPU work from off-CPU waiting (network, disk, subprocess, lock contention).

Existing pieces this effort builds on:

FBUILD_PERF_LOG=1 env-gated phase timing in crates/fbuild-build/src/perf_log.rs (from perf(build): investigate warm-pass compilation stall — 30s where cache says <1s #91) — coarse per-phase wall clock, emitted via tracing + stderr.
fbuild-packages already has a parallel download pipeline; unknown how well it overlaps download → extract → install → compile in practice.
Embedded zccache service (perf(zccache): migrate from managed wrapper binary to embedded ZccacheService in fbuild-daemon #789) covers compiler-invocation caching; this effort targets everything around the compiler.
The NightDriverStrip benchmark fixture is referenced by the ignored test build_nightdriverstrip_demo in crates/fbuild-build/tests/esp32_build.rs (expects tests/NightDriverStrip/); the checkout is not committed, so the Docker harness must clone it.
zackees/soldr has the Docker + script reference infrastructure (docker/cook-shared-cache/Dockerfile, perf/, PERF.md) to model the fast-rebuild container on.

Proposal

Stand up a reproducible Linux Docker profiling harness, measure cold-cache and hot-cache builds of NightDriverStrip, and use the data to drive a burndown of optimization sub-issues until fbuild's non-compiler overhead is as close to fully overlapped/concurrent as possible.

Phase 0 — Docker profiling harness

Dockerfile optimized for fast image rebuilds (layer-cached toolchain/deps, source COPY last, or bind-mount + named volumes per the soldr pattern), based on zackees/soldr's docker + script infrastructure.
fbuild's own cache (~/.fbuild/cache) is not persisted between container runs — cold cache means genuinely cold (fresh downloads).
Clones NightDriverStrip as the benchmark workload (ESP32 demo env; optionally demo_c6 for a RISC-V data point).
Orchestration script runs: (a) cold-cache build, (b) hot-cache rebuild (same container, cache intact), each N≥3 times, and archives all profiles/logs as artifacts.

Phase 1 — Instrumentation + profiling

Event logging on: FBUILD_PERF_LOG=1 plus extending perf_log.rs/tracing spans wherever coverage is missing (download start/end per package, extract, install, per-TU compile dispatch, archive, link, daemon RPC round-trips).
On-CPU profiling: perf record -g (or samply/cargo flamegraph) over daemon + CLI → flamegraphs.
Off-CPU (async) profiling: off-CPU flamegraphs (perf sched / eBPF offcputime) and/or tokio-console-style async task instrumentation to expose where the pipeline waits — network, disk, subprocess, serialized stages, lock contention.

Phase 2 — Analysis → sub-issues

Produce a cold-vs-hot phase breakdown table (wall clock per stage, % of total).
For every stage that is (a) not cached when it should be, (b) serialized when it could overlap with download/install/compile/link, or (c) hot on-CPU in fbuild's own code: file a child sub-issue with the profile evidence attached.

Phase 3 — Optimization burndown

Expected themes (to be confirmed by data, not assumed): overlap download ⇄ extract ⇄ install ⇄ first compiles; start compiling TUs whose deps are ready before the full install finishes; overlap archive/link prep with trailing compiles; cache anything recomputed on hot builds (config parse, library selection, header scan); remove sync-in-async stalls (continuing audit: sync code that could be async in fbuild-cli + fbuild-python (sub-issue of #813) #817).
Out of scope: compile settings. No changes to compiler/linker flags, optimization levels, or codegen — stock settings stay stock. Everything around the compiler is fair game.
Each optimization ships as its own PR against its sub-issue, with before/after numbers from the Phase 0 harness in the PR description.

Phase 4 — Verification + close-out

Re-run the harness after each merged PR; final report of cold and hot wall-clock deltas vs the Phase 1 baseline on this issue.

Acceptance criteria

Docker harness merged (Dockerfile + run script) that produces cold-cache and hot-cache builds of NightDriverStrip with fbuild cache not persisted between docker runs.
On-CPU and off-CPU profiles + FBUILD_PERF_LOG event timelines captured for both cold and hot runs and attached to this issue.
Baseline numbers posted: cold and hot wall clock with per-phase breakdown (median of ≥3 runs).
Every identified slow/uncached/serialized path has a child sub-issue linked from a task list on this issue.
All child sub-issues resolved via merged PRs, each PR showing before/after harness numbers.
Final cold + hot wall-clock comparison vs baseline posted; all sub-issues closed; this issue closed.

Decisions

Benchmark workload: NightDriverStrip demo env (ESP32/Xtensa) — it's the fixture the existing ignored integration test already targets and the heaviest real-world project we've built.
Fixture provisioning: Docker harness clones NightDriverStrip at a pinned commit rather than committing the tree to this repo — keeps the repo lean, keeps runs reproducible.
Profiler choice: perf + flamegraphs for on-CPU, off-CPU flamegraphs (perf sched/eBPF) for waits — standard Linux tooling that works in a container; exact tool swap is fine if the harness PR finds something better.
Harness location: ci/docker-profile/ alongside the existing ci/docker-* dirs; scripts in Python via uv run per the language policy (CI scripting is the sanctioned Python use).
Sub-issue granularity: one issue per independently-mergeable optimization, all linked from a task list here — matches the meta-issue pattern used by meta: broker-safe global artifact repository feature burndown #603.
Priority: P2 — significant DX win, nothing shipping is blocked on it.
Compile settings frozen: stock flags only; performance must come from caching, concurrency, and pipeline overlap.

Related issues

perf(build): investigate warm-pass compilation stall — 30s where cache says <1s #91 — warm-build phase timing (FBUILD_PERF_LOG) — this effort extends that instrumentation.
audit: sync code that could be async in fbuild-cli + fbuild-python (sub-issue of #813) #817 — sync-code-that-could-be-async audit — Phase 3 continues that thread with profile evidence.
perf(zccache): migrate from managed wrapper binary to embedded ZccacheService in fbuild-daemon #789 — embedded zccache service — covers compiler caching; explicitly out of scope here.

Burndown (Phase 2 task list)

Correctness blockers found by the harness (done):

fix(build): lite-scons harness JSON protocol is corrupted by user-script print() #945 — extra_scripts stdout corrupts lite-SCons JSON protocol (merged: fix(build): seal lite-scons harness JSON channel against user-script stdout #946)
fix(build): escaped quotes in build_flags defines are mangled before reaching gcc (ESP32 sketch compile) #947 — escaped-quote defines mangled before gcc (merged: fix(config): consume INI shell-quoting layer in parse_flags #949)
CI unbreak: BoardConfig: Default + rustdoc link from fix(release): restore macOS + arm64-windows lanes on soldr 0.7.98 + setup-soldr v0.9.64 #941 (merged: fix(config): derive Default for BoardConfig (unbreaks workspace check) #948)

Harness:

feat(ci): Docker Linux profiling harness for NightDriverStrip builds #950 — Docker profiling harness (ci/docker-profile/)

Optimization sub-issues (from the Phase-1 baseline):

perf(build): no-change rebuild recompiles the entire sketch + core variant (~103s; should be seconds) #951 — no-change rebuild recompiles sketch + core variant (fixed: 108 s → 2.1 s steady-state, PR fix(build): stop no-change rebuilds from recompiling every TU #956)
perf(build): framework libs (~150s) recompiled per project; core cache never hydrates #952 — fw-libs recompiled per project (~150s); root-caused (fw-libs compiles miss zccache: absolute paths + cwd=None). Fix plan posted; impl pending. Docs from this work: docs(agents): path prefix-root conventions + cross-project cache-key pitfalls #961
[~] perf(packages): overlap download/extract/install in cold pioarduino-resolve (~185-237s serialized) #953 — overlap cold download/install: toolchain ∥ framework+SDK now concurrent (PR perf(esp32): download GCC toolchain concurrently with framework + SDK libs (#953) #967); further overlap (lib_deps ∥ resolve, compile ∥ extract) deferred
feat(deploy): manage esptool as a provisioned package instead of requiring pip install #954 — manage esptool as a provisioned package (P3)
fix(build): project-relative board_build.partitions CSV not found (resolved against framework tools dir) #955 — project-relative board_build.partitions CSV ignored (fixed: PR fix(build): resolve board_build.partitions CSV project-relative first #958)
perf(build): first rebuild after a cold build recompiles once (signature drift between cold and steady state) #957 — first-rebuild-after-cold recompile FIXED (depfile-own-mtime false-stale, PR fix(build): don't treat depfile's own mtime as stale → no more first-rebuild recompile (#957) #971 merged); build_unflags variant → fix(build): rebuild_signature omits build_unflags stripping → persistent recompiles for build_unflags projects #970
build(esp32): NightDriverStrip partitions CSV rejected by esp32-core-3.3.9 gen_esp32part.py (nvs 0x2000 < 0x3000) #960 — NightDriverStrip partitions CSV rejected by gen_esp32part.py / framework-parity (follow-up to fix(build): project-relative board_build.partitions CSV not found (resolved against framework tools dir) #955)

fix(build): seal lite-scons harness JSON channel against user-script …

fix(config): consume INI shell-quoting layer in parse_flags

fix(config): consume INI shell-quoting layer in parse_flags (#949)

fix(ci): record perf data to container-local scratch, not the 9P bind…

Phase 1 baseline (harness: PR #950, fbuild @ main with #946 + #948 + #949)

Environment: Docker Desktop Linux container (WSL2 kernel 6.18), 4 vCPU / 8 GB, NightDriverStrip @ 5911331 env:demo, FBUILD_PERF_LOG=1, on-CPU perf (cpu-clock, −F 99, fp stacks) + off-CPU bpftrace/sched running during every build. fbuild cache wiped per cold run; 3 iterations per scenario.

Wall clock (median of 3)

scenario	median	min	max
cold (empty `~/.fbuild` + `~/.platformio`, fresh project)	774.2 s	700.8	808.4
warm (warm global cache, fresh project copy)	435.3 s	429.9	447.8
hot (no-change rebuild)	107.8 s	103.7	121.2

Per-phase medians (`FBUILD_PERF_LOG`, esp32-orchestrator)

phase	cold	warm	hot
pioarduino-resolve	236.7 s	0.02 s	0.01 s
fw-libs	162.6 s	150.7 s	0.001 s
compile-core-variant	11.0 s	11.0 s	10.7 s
compile-sketch	85.7 s	101.0 s	92.0 s
link-convert-size	5.1 s	4.7 s	4.4 s
config-parse	0.4 s	0.4 s	0.5 s

On-CPU profile (cold): cc1plus dominates as expected (real compiler work — out of scope per the stock-compile-settings constraint); fbuild's own tokio-rt-worker CPU is ~9% of compiler CPU. The wall-clock gaps above are therefore cache/pipelining problems, not fbuild CPU problems.

Findings → sub-issues

A no-change rebuild takes ~108 s — compile-sketch (92 s) and compile-core-variant (10.7 s) re-run every build; only fw-libs short-circuits. The embedded zccache also doesn't convert these recompiles into hits. Target: ≤ a few seconds.
fw-libs (~150 s) is recompiled for every fresh project with an identical platform/flags, and core-cache-hydrate never hits (0 ms in all 9 runs while compile-core-variant always runs ~11 s). Framework/core caching should be global, like the disk cache for packages.
Cold pioarduino-resolve is 185–237 s of mostly serialized download → extract → install (stderr shows repeated sequential downloading esp32-xtensa-gcc lines). Downloads, extraction, and installs should overlap with each other and with early compile phases.
esptool is not a managed package — a cold build on a clean machine fails at elf2image with "Install with: pip install esptool" (the harness now bundles it; fbuild should provision it like other tools).

Bugs already found and fixed by the harness on the way here: #945 (extra_scripts stdout corrupted the lite-SCons JSON protocol) and #947 (escaped-quote defines mangled before gcc) — both merged, NightDriverStrip demo now builds end-to-end (firmware.bin 753 KB, flash 18.4 %).

Artifacts (timings.jsonl, per-run flamegraphs, folded stacks, daemon perf-log lines) are archived locally under ci/docker-profile/out/baseline1/; regeneration is one command once #950 lands: uv run python ci/docker-profile/run_profile.py.

feat(ci): Docker Linux profiling harness for NightDriverStrip builds (#…

fix(build): stop no-change rebuilds from recompiling every TU (#956)

feat(dylint): ban raw Path::{starts_with, strip_prefix} in production…

perf(esp32): download GCC toolchain concurrently with framework + SDK…

fix(build): don't treat the depfile's own mtime as a stale prerequisi…

Phase 4 — Final report & close-out

Closing this P2 tracking issue: the core burndown is delivered — the Docker profiling harness, cold/hot profiling, the baseline breakdown, and the significant optimizations the profile surfaced. The remaining items are genuinely cross-repo, CI-infra, or P3, and each now lives as its own independently-tracked issue (listed below) rather than blocking this meta indefinitely.

What shipped (15 PRs merged)

Harness + instrumentation (Phase 0–1)

feat(ci): Docker Linux profiling harness for NightDriverStrip builds #950 — ci/docker-profile/ Docker Linux harness (perf on-CPU + bpftrace off-CPU + FBUILD_PERF_LOG timelines; fbuild cache not persisted between runs; pinned NightDriverStrip). This is what drove every finding below.

Headline optimization

perf(build): no-change rebuild recompiles the entire sketch + core variant (~103s; should be seconds) #951 / fix(build): stop no-change rebuilds from recompiling every TU #956 — no-change rebuild recompiled the whole sketch + core variant every time. Root cause: per-TU staleness asymmetry (depfile prereq cwd + rebuild_signature unflag/relativization). Steady-state no-op rebuild 108 s → 2.1 s — the reproducible, dominant dev-loop win.
perf(build): first rebuild after a cold build recompiles once (signature drift between cold and steady state) #957 (fix(build): don't treat depfile's own mtime as stale → no more first-rebuild recompile (#957) #971) — the first rebuild after a cold build recompiled every TU once: dependency_is_newer_than_object compared the object against its own .d (an output of the same gcc invocation, always newer). Removed the false-stale check + regression tests. First-rebuild-after-cold is now a true no-op.

Cold-pipeline concurrency (Phase 3 theme: overlap download/install)

perf(esp32): download GCC toolchain concurrently with framework + SDK libs (#953) #967 / fix(esp32): surface both errors if toolchain AND framework resolve fail (#967 follow-up) #969 — the GCC toolchain (~100 MB) now downloads concurrently with the framework + SDK libs (~hundreds MB) instead of back-to-back; framework→libs ordering preserved.
perf(build): compile ESP32 framework libs workspace-relative (cross-project cache stability, #952) #965 — ESP32 framework libs now compile workspace-relative (like sketch/core), so their zccache keys are project-portable. Warm fw-libs 185 s → ~141 s. The full sub-second win is gated on cross-project zccache keying (see perf(zccache): firmware compiles don't hit zccache cross-project even when args are workspace-relative #966).

Correctness blockers the harness exposed (were silently breaking real builds)

fix(build): seal lite-scons harness JSON channel against user-script stdout #946 (fix(build): lite-scons harness JSON protocol is corrupted by user-script print() #945 lite-SCons JSON stdout corruption), fix(config): consume INI shell-quoting layer in parse_flags #949 (fix(build): escaped quotes in build_flags defines are mangled before reaching gcc (ESP32 sketch compile) #947 escaped-quote defines), fix(config): derive Default for BoardConfig (unbreaks workspace check) #948 (BoardConfig: Default CI unbreak), fix(build): resolve board_build.partitions CSV project-relative first #958 (fix(build): project-relative board_build.partitions CSV not found (resolved against framework tools dir) #955 project-relative partitions CSV).

Guardrails / docs / repo hygiene

docs(agents): path prefix-root conventions + cross-project cache-key pitfalls #961 (path prefix-root conventions doc), feat(dylint): ban raw Path::{starts_with, strip_prefix} so the #952 cache-key mistake can't recur #963 (ban_raw_path_prefix_compare dylint), style(teensy): rustfmt flash.rs — unblock the Formatting CI gate #972 (restored the repo-wide Formatting CI gate that had been red on main since fix(teensy): run flash in build dir + sweep stray dash-flag files (no more -r in repo root) #968).

Measurement notes (honest)

The no-op rebuild 108 s → 2.1 s delta is rock-solid and reproducible.
Cold total wall-clock is dominated by cc1plus compiling the framework/FastLED/sketch TUs — that's compile work, explicitly out of scope ("stock flags only"). The harness confirmed non-compiler overhead (download/extract/install/resolve) is the minority of cold time and is now partially overlapped (perf(esp32): download GCC toolchain concurrently with framework + SDK libs (#953) #967).
Single-run cold/warm comparisons in the harness carry real network + CPU-contention variance; where a delta was noisy I've said so on the relevant PR rather than overclaim.

Remaining work — rolled up into round-2 meta #974

The residual items are cross-repo, CI-infra, or P3 — none blocks the value delivered above. They're tracked together in the follow-on meta #974 (with root cause + fix path for each), so this P2 meta doesn't stay open indefinitely:

perf(zccache): firmware compiles don't hit zccache cross-project even when args are workspace-relative #966 — firmware compiles don't hit zccache cross-project; the real warm-cache win (cross-repo zccache worktree_root change). Supersedes the remainder of perf(build): framework libs (~150s) recompiled per project; core cache never hydrates #952.
ci(dylint): Dylint job is red on main — alias-loop workaround fails, so NO custom lints are enforced #964 — Dylint CI red on main (dylint-link not wired as the link-time linker).
fix(build): rebuild_signature omits build_unflags stripping → persistent recompiles for build_unflags projects #970 — latent build_unflags rebuild_signature asymmetry.
perf(packages): overlap download/extract/install in cold pioarduino-resolve (~185-237s serialized) #953 — further cold overlap (lib_deps ∥ resolve; compile ∥ SDK-extract) beyond the merged toolchain∥framework overlap.
feat(deploy): manage esptool as a provisioned package instead of requiring pip install #954 — provision esptool as a managed package (P3).
build(esp32): NightDriverStrip partitions CSV rejected by esp32-core-3.3.9 gen_esp32part.py (nvs 0x2000 < 0x3000) #960 — partitions CSV framework-parity (P3, follow-up to fix(build): project-relative board_build.partitions CSV not found (resolved against framework tools dir) #955).

Acceptance-criteria status

Docker harness merged (cache not persisted between runs) — feat(ci): Docker Linux profiling harness for NightDriverStrip builds #950
On-CPU + off-CPU profiles + FBUILD_PERF_LOG timelines captured (cold + hot)
Baseline per-phase breakdown posted
Every identified slow/uncached/serialized path filed as a linked child sub-issue
[~] Child sub-issues: the correctness + major-optimization set is merged; the cross-repo/infra/P3 remainder is re-scoped to the independent issues above
Final cold + hot deltas posted (this comment)

Closing per maintainer decision: core profiling-driven burndown delivered; residual refinements tracked independently.

Closing per maintainer decision (P2 tracking issue): core profiling-driven burndown delivered — harness, profiling, baseline, and the major optimizations (15 PRs merged). Residual cross-repo/infra/P3 items rolled up into #974.

fix(build): make ESP32 sketch+core compiles hit zccache cross-project (…

fix(build): route framework-lib compiles through embedded zccache (#986…

Navigation Menu

Repository navigation

meta: profiling-driven build performance burndown (Docker Linux, NightDriverStrip, cold vs hot cache) #942

Context

Proposal

Phase 0 — Docker profiling harness

Phase 1 — Instrumentation + profiling

Phase 2 — Analysis → sub-issues

Phase 3 — Optimization burndown

Phase 4 — Verification + close-out

Acceptance criteria

Decisions

Related issues

Burndown (Phase 2 task list)

Phase 1 baseline (harness: PR #950, fbuild @ main with #946 + #948 + #949)

Wall clock (median of 3)

Per-phase medians (`FBUILD_PERF_LOG`, esp32-orchestrator)

Findings → sub-issues

Phase 4 — Final report & close-out

What shipped (15 PRs merged)

Measurement notes (honest)

Remaining work — rolled up into round-2 meta #974

Acceptance-criteria status

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

meta: profiling-driven build performance burndown (Docker Linux, NightDriverStrip, cold vs hot cache) #942

Description

Context

Proposal

Phase 0 — Docker profiling harness

Phase 1 — Instrumentation + profiling

Phase 2 — Analysis → sub-issues

Phase 3 — Optimization burndown

Phase 4 — Verification + close-out

Acceptance criteria

Decisions

Related issues

Burndown (Phase 2 task list)

Activity

zackees commented on Jul 2, 2026

Phase 1 baseline (harness: PR #950, fbuild @ main with #946 + #948 + #949)

Wall clock (median of 3)

Per-phase medians (FBUILD_PERF_LOG, esp32-orchestrator)

Findings → sub-issues

zackees commented on Jul 5, 2026

Phase 4 — Final report & close-out

What shipped (15 PRs merged)

Measurement notes (honest)

Remaining work — rolled up into round-2 meta #974

Acceptance-criteria status

zackees commented on Jul 5, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Per-phase medians (`FBUILD_PERF_LOG`, esp32-orchestrator)