fix(bench): align ESP32-S3 toolchains and explain cold-build gap #1489

Context

The first published ESP32-S3 Blink comparison (run 36273077249, SHA 8cd955adc66cb8619faa50fbdc1b71fecccea3ca, latest.json) reports these medians from three trials on ubuntu-24.04 (nproc=4):

The cold fbuild build is 2.45× slower, while its no-change rebuild is 3.97× faster. The three cold fbuild trials are tightly grouped (16.01–16.35 s), and daemon_restarts is zero.

The benchmark log artifact shows where most of the cold time goes: after fbuild clean cache, fbuild reports a framework-core cache miss and recompiles 60 core files, reaching 20/40/60 at about 4.4/9.5/13.3 s, then compiles the sketch and finishes at about 16 s. The first PlatformIO cold trial logs 46 FrameworkArduino compile commands and finishes in about 6.4 s. Both compile the same Blink sketch, but they do not use the same ESP32 build inputs:

• bench/blink/platformio.ini pins espressif32@6.13.0. PlatformIO resolves framework-arduinoespressif32 @ 3.20017.241212+sha.dcc1105b (Arduino-ESP32 2.0.17 generation) and toolchain-xtensa-esp32s3 @ 8.4.0+2021r2-patch5.
• Every timed fbuild build warns that it ignores that registry version pin and uses its own pinned ESP32 packages. Its log reports xtensa-esp-elf-gcc 14.2.0. In code, Esp32Platform::new defaults to the pioarduino stable release, which resolves its framework/toolchain from platform.json. The benchmark does not record the actual fbuild framework version.
• The workflow's ARDUINO_ESP32_CORE_VERSION=3.3.7 pins Arduino CLI only; it does not align PlatformIO and fbuild. See workflow, fixture, fbuild package resolution, and default platform.

This identifies the immediate reason the chart is misleading: it compares different framework generations, compilers, and core source sets. It also identifies the dominant fbuild time as cold framework-core compilation. The log alone cannot quantify how much of the 9.5 s gap comes from source count, compiler version, flags, cache overhead, or linking; a controlled same-stack run is required before attributing all of it to fbuild's implementation.

The current harness hides this ESP32 result in two ways: PERF_PHASE_LABELS contains only avr-orchestrator and pipeline, so the published ESP32 cold_phases_ms is empty; fbuild_vs_platformio_cold and fbuild_overhead_ms select the first board (Uno), so the top-level ratio is 0.470 even while ESP32-S3 is 2.449. The raw compiler floor is likewise collected only for Uno. See runner.

Proposal

Make the ESP32-S3 comparison reproducible and like-for-like: resolve the fixture's espressif32@6.13.0 PlatformIO alias through #1491, then build PlatformIO and fbuild against the same immutable Arduino core, ESP-IDF and Xtensa toolchain package set. Cross-version labeling is an interim safety measure, not the final fix for a valid alias. Record resolved package/compiler versions in the artifact. Add ESP32 phase timing and a per-board raw-compiler baseline, then run matched cold/warm trials to separate framework size, compiler work, cache overhead, and link/image time. Optimize the measured dominant fbuild phase if the same-stack cold result remains slower.

Make ratios and regression warnings board-specific so an ESP32-S3 cold regression cannot be masked by Uno's result.

Acceptance criteria

• RED: a focused benchmark-runner test demonstrates that mismatched resolved ESP32 packages can currently be published as directly comparable, and that the current top-level cold ratio selects Uno while ESP32-S3 is slower.
• GREEN: after the valid espressif32@6.13.0 alias is supported, the workflow runs PlatformIO and fbuild with the matching resolved ESP32 framework/toolchain/SDK inputs and records their identities. Until then, it labels or excludes cross-version results; labeling alone does not satisfy the final matched-stack benchmark.
• ESP32-S3 has nonempty per-phase cold timings, a raw compile baseline for its actual source set, and a board-specific fbuild/PlatformIO cold ratio in latest.json and history. Regression reporting evaluates that board-specific ratio.
• A fresh three-trial run records the ESP32-S3 core source counts, compile flags/toolchain provenance and cold phase breakdown, then documents which phase accounts for the remaining cold gap under matched inputs. If a code change addresses that phase, the focused before/after repro is RED → GREEN and the no-change warm result does not regress.
• Unit tests: feed an offline espressif32@6.13.0 registry/platform-manifest fixture through the shared fbuild-core resolver and assert the exact framework/toolchain payload identities; separately assert the benchmark runner flags any requested-versus-resolved stack mismatch. Keep the three-trial build benchmark as an integration check.

PlatformIO alias dependency

• This benchmark must exercise actual espressif32@6.13.0 resolution, not a server-side alias to pioarduino stable or an early refusal of the valid pin. meta: resolve PlatformIO package aliases and build requested stacks #1491 owns the shared resolver; ESP32: provision per-MCU registry toolchains so pre-unified platform pins (pioarduino 51.x) actually build against their pin #1444 owns legacy per-MCU toolchain support needed by older ESP32 stacks.

Decisions

• Treat this as a benchmark-validity and ESP32 cold-build performance bug, priority P2: the published comparison currently implies a tool-performance conclusion that the differing package stacks cannot support.
• Keep the same Blink fixture and cold policy so the corrected run can be compared with the published one; change cache treatment only if applied consistently to both tools.
• Do not infer that GCC 14 alone explains the gap. The current evidence proves a stack mismatch and locates most fbuild time in its core compile, but does not isolate the relative contributions.

Related issues

• feat(bench): publish ESP32-S3 build timings alongside Arduino Uno #1488 added the ESP32-S3 benchmark; this issue investigates its first published result.
• meta: cold builds 50% faster than PlatformIO (performance audit) #1469 tracks the Uno cold-build performance audit and the broader benchmark fairness policy; it does not cover ESP32-S3's package mismatch.

Core generic-path testing gate

• RED → GREEN: resolve the benchmark fixture’s platformio/platform/espressif32@6.13.0 through the public fbuild-core PlatformIO-path entry point, not through benchmark-only or ESP32-only lookup. Offline fixtures assert the exact platform payload URL/SHA-256, host selection, cache identity, and manifest-derived framework, SDK, and toolchain identities; no live registry or compiler download is required.
• Include a synthetic valid platform path absent from native dispatch in the shared core matrix to prove generic payload resolution. Keep benchmark-runner tests separate: they must flag requested-versus-resolved stack mismatches and only publish matched-stack comparisons as like-for-like.
• The benchmark integration run then verifies that fbuild consumes the core-resolved stack and records its identities; merely labeling a mismatched stack is an interim safeguard, not completion.