fbuild build: add --shrink[=MODE] / --no-shrink for aggressive flash reduction #492

Motivation

A stock Arduino-ESP32 3.3.7 build of tests/platform/esp32s3 on this branch lands at 329,776 B flash, of which ~30 KB is newlib stdio that we could replace with picolibc's tinystdio while keeping %f/%lld/%e/%g support. Measured fan-out on the actual ELF:

Replacement cost (picolibc tinystdio __d_vfprintf, _NEED_IO_LONG_LONG=1, _NEED_IO_DOUBLE=1, io-float-exact=false, no C99/positional/wchar): ~5.5–7 KB flash on xtensa-esp-elf (extrapolated from picolibc upstream Cortex-M7 numbers in printf.md, +15% for xtensa density).

Net achievable on this sketch: ~22–24 KB flash + ~312 B RAM (~11% of attributed code).

This issue proposes a CLI knob — --shrink[=MODE] / --no-shrink — that exposes this and several adjacent wins behind one named flag, with auto-detection that picks the right default per framework + IDF version. Picolibc is the upstream direction (ESP-IDF 6.0 already defaults to it), so the wrap auto-noops on future toolchains via #ifdef __PICOLIBC__.

Prior art: ESPHome PR #16170 (the picolibc cookie-FILE wrap that informed this design) and ESPHome PR #16189 (the IDF-6-disables-it pattern).

Proposed flag surface

On the Build, Deploy, TestEmu, and Ci subcommands:

// crates/fbuild-cli/src/cli/args.rs

/// Aggressive flash-size reduction. MODE selects how hard fbuild tries:
///   auto        (default) decide from framework + IDF version
///   off         no shrinking; use stock libc, full symbol set
///   safe        no-behavior-change wins only (printf-thin wrap)
///   aggressive  safe + behavior-altering (strip esp_err_msg, disable coredump)
///   printf      single-knob: only the printf-family wrap (debuggable A/B)
#[arg(long, value_enum, num_args = 0..=1, default_missing_value = "auto")]
pub shrink: Option<ShrinkMode>,

/// Disable all shrink optimizations. Equivalent to --shrink=off.
/// Convenience for quick A/B comparisons; always wins over --shrink.
#[arg(long = "no-shrink", conflicts_with = "shrink")]
pub no_shrink: bool,

#[derive(clap::ValueEnum, Clone, Copy, Debug, Default, PartialEq, Eq)]
pub enum ShrinkMode { #[default] Auto, Off, Safe, Aggressive, Printf }

--shrink with no value resolves to auto. --no-shrink overrides any auto-enable from project config or framework detection.

Mode taxonomy

Auto-resolution matrix

Libc detection runs after the toolchain probe — fbuild already inspects libc.a; we add a single check for whether __PICOLIBC__ is defined by the libc headers. The resolved decision is always printed (see reporting below) so behavior is never silently auto-enabled.

Six concrete invocations

# Default — let fbuild decide
fbuild build tests/platform/esp32s3 -e esp32s3
# → Shrink: auto → safe (libc=newlib, framework=arduino-esp32@3.3.7)

# Explicit on, default mode
fbuild build tests/platform/esp32s3 -e esp32s3 --shrink

# Maximum, including sdkconfig knobs
fbuild build tests/platform/esp32s3 -e esp32s3 --shrink=aggressive

# Just the printf swap (debugging the swap in isolation)
fbuild build tests/platform/esp32s3 -e esp32s3 --shrink=printf

# Hard off — reproduce stock newlib build for A/B baseline
fbuild build tests/platform/esp32s3 -e esp32s3 --no-shrink
# → Shrink: --no-shrink (user override; would otherwise default to safe)

# Equivalent explicit form
fbuild build tests/platform/esp32s3 -e esp32s3 --shrink=off

Build-output reporting

Default case (--shrink=auto, the no-flag default): when the platform's auto-resolver decides to apply shrinks, fbuild prints a single concise line in green listing the symbol families being wrapped. When the platform has no auto-shrink to apply (e.g. AVR, or ESP-IDF 6.x where picolibc is already the libc), the line is omitted entirely — silence is the default.

For ESP32-S3 / Arduino-ESP32 IDF 5.x (auto-resolver picks safe, which enables the printf-thin wrap):

0.00 ========================
0.00 =   BUILDING esp32s3   =
0.00 ========================
0.11 Toolchain: xtensa-esp-elf-gcc 14.2.0
0.12 auto shrinking: vfprintf, vsnprintf, snprintf, printf, vprintf, fprintf, vsprintf, sprintf   <- green
...
341.48 Flash:  306.05KB / 8.00MB (3.7%)

For AVR / megaavr (no auto-shrink available — avr-libc already minimal):

0.00 ========================
0.00 =   BUILDING uno   =
0.00 ========================
0.11 Toolchain: avr-gcc 7.3.0
   <- no shrink line; silently skipped
...

For ESP-IDF 6.x (libc is already picolibc — wrap would be net-negative, auto resolves to off):

0.12   <- no shrink line; auto picked off

Explicit modes (--shrink=safe|aggressive|printf|off or --no-shrink): print the verbose plan so the user can see exactly what fired and what was skipped — explicit intent earns explicit reporting:

0.12 Shrink: aggressive (explicit)
0.12   [x] printf-thin (picolibc tinystdio wrap, est. -24 KB flash)
0.12   [x] esp_err_msg_table strip (est. -1.8 KB RAM)
0.12   [x] coredump strip (est. -3.0 KB flash, -1.9 KB RAM)
0.12   [x] C++ exception personality drop (est. -0.7 KB flash)
...
341.48 Flash:  302.10KB / 8.00MB (3.7%)  [stock: 329.78KB; saved 27.7 KB]
341.48 RAM:    665.50KB / 320.00KB (208.0%) [stock: 670.25KB; saved 4.8 KB]
341.48 Shrink result: flash -27.7 KB (-8.4 %), ram -4.8 KB (-0.7 %)

0.12 Shrink: --no-shrink (user override; auto would have applied: printf-thin)

0.12 Shrink: off (explicit)

The "stock" comparison line only prints if we have a cached baseline from a previous --no-shrink build of the same env. Otherwise it is suppressed.

Reporting rules summary

Color / styling

Use owo-colors = "4" (tiny, auto-disables on non-tty and when NO_COLOR is set). The auto shrinking: line is green(); the prefix label auto shrinking: is green().bold(); the comma-separated symbol list is plain green(). The verbose plan uses default color with a single bold first line.

Per-platform auto-shrink registry

The categories printed in the green line are sourced from a per-platform registry in crates/fbuild-build/src/shrink/registry.rs:

pub struct AutoShrinkEntry {
    pub category: &'static str,   // "printf-thin", "esp_err_msg", etc.
    pub symbols: &'static [&'static str],  // printed in the green one-liner
}

// Example registry data:
// arduino-esp32 (IDF 5.x newlib): [
//   AutoShrinkEntry { category: "printf-thin",
//     symbols: &["vfprintf", "vsnprintf", "snprintf", "printf",
//                "vprintf", "fprintf", "vsprintf", "sprintf"] },
// ],
// arduino-esp32 (IDF 6.x picolibc): [],   // empty -> no green line
// avr / megaavr:                    [],   // empty -> no green line
// stm32 / nrf52 / rp2040 / teensy:  [],   // empty in v1

When a new platform shrinker is added, it appends to its platform's registry entry and the symbols automatically show up in the green one-liner — no further reporting plumbing needed.

platformio.ini integration

[env:esp32s3]
platform = espressif32
board = esp32-s3-devkitc-1
framework = arduino
fbuild_shrink = aggressive       ; off | safe | aggressive | printf | auto
fbuild_shrink_skip = coredump    ; optional escape hatch, comma-separated

Precedence (lowest -> highest): platform default -> platformio.ini -> CLI flag. --no-shrink always wins.

A/B workflow this enables

fbuild build tests/platform/esp32s3 -e esp32s3 --no-shrink \
    --output-dir /tmp/baseline
fbuild build tests/platform/esp32s3 -e esp32s3 --shrink=aggressive \
    --output-dir /tmp/shrunk
fbuild bloat diff /tmp/baseline/firmware.elf /tmp/shrunk/firmware.elf
# (a future `bloat diff` subcommand, separately tracked)

Linking strategy: spec-file primary, --wrap fallback

Two architectural options exist for redirecting vfprintf calls from newlib to vendored picolibc; we adopt the spec-file path as primary.

Primary: GCC spec file (printf-thin.specs)

fbuild emits ~/.fbuild/cache/printf-thin/<toolchain>/printf-thin.specs at first build and adds --specs=<path>/printf-thin.specs to the link line. The spec rewrites the *libc: rule to insert -lprintf_thin before -lc:

*libc:
%{!shared-libgcc:--start-group -L${printf_thin_dir} -lprintf_thin -lc -lnosys --end-group}

libprintf_thin.a is a complete shadow of newlib's vfprintf.o + svfprintf.o co-exports: it defines vfprintf, _vfprintf_r, _printf_r, printf, __sbprintf, _svfprintf_r, vsnprintf, _vsnprintf_r, etc. — every symbol newlib pulls those two objects in for. Linker resolves all of them to our archive in the --start-group scan, never pulls newlib's vfprintf.o / svfprintf.o. No archive surgery, no toolchain modification.

Why this is the right primary path:

• Naturally LTO-safe. Our archive is precompiled native code (-fno-lto). Sketch bitcode references vfprintf as external; LTO has no bitcode definition to inline; the reference survives to the final link and resolves to our archive. None of the --wrap-under-LTO gymnastics needed.
• Clean stack traces. Frames show vfprintf (the picolibc one), addr2line resolves to vendored picolibc source. No __wrap_* confusion.
• Composes with --specs=nano.specs. Users who already pass nano.specs and the user-passes-our-specs get both — our printf-thin wins because our -L is first, and the rest of nano (smaller strtol, etc.) still applies. We get a free win that's hard to express with --wrap.
• Mirrors how Espressif, ARM, and Newlib itself ship printf variants (nano.specs, picolibc.specs). Linker behavior is well-understood; users debugging link issues find familiar territory.

Cost: a per-newlib-version symbol manifest (~50 lines listing each printf-related symbol newlib co-exports from vfprintf.o/svfprintf.o). Newlib's stdio layout has been stable for ~15 years; manifest changes once every few years and is covered by a CI test that diffs nm output against the manifest.

Fallback: -Wl,--wrap= per-symbol

When the spec-file path can't be used (toolchain without spec-file support, or the resolved auto path is printf mode for debugging the wrap in isolation), fall back to the ESPHome-style wrap:

-Wl,--wrap=vfprintf,--wrap=vprintf,--wrap=printf,--wrap=fprintf
-Wl,--wrap=vsnprintf,--wrap=snprintf,--wrap=vsprintf,--wrap=sprintf
-Wl,--wrap=fiprintf,--wrap=vfiprintf            ## for __assert_func, per review comment

LTO safety here requires two belt-and-suspenders steps:

1. Compile the vendored picolibc TUs with -fno-lto. LTO can't inline across the native/bitcode boundary; the vfprintf reference survives to the linker where --wrap rewrites it.
2. Mark the stub functions __attribute__((noinline, externally_visible)). The externally_visible attribute lives in the bitcode and tells lto1 "this symbol is referenced from outside the LTO universe; don't IPA-eliminate it."

Modern GCC's linker plugin (≥ 4.6) communicates --wrap resolutions to lto1, so wrapped symbols are treated as preempted in most cases. The two steps above make the behavior independent of plugin version.

Linker scripts cannot fix this because they run in step 5 of the LTO pipeline — after lto1 has already done IPA inlining in step 4. By the time KEEP() / PROVIDE() / EXCLUDE_FILE() apply, any inlined vfprintf references are gone. The spec-file approach sidesteps the issue entirely by keeping our printf out of the LTO unit.

Comparison table

Implementation sketch

1. crates/fbuild-build/src/shrink/mod.rs — ShrinkMode, ShrinkPlan { applied: Vec<Applier>, skipped: Vec<(Applier, Reason)> }, auto-resolver. Fingerprint augmentation: include ShrinkMode + resolved applier set in build_fingerprint/mod.rs so toggling --shrink invalidates the link cache (per review comment).
2. crates/fbuild-build/src/shrink/printf_thin.rs — vendored picolibc tinystdio (~8 files from newlib/libc/tinystdio/: vfprintf.c, vfscanf.c, dtoa_engine.c, dtoa.h, stdio_private.h, formatter-spec headers; BSD-3-clause, ship NOTICE + COPYING.picolibc + COPYING.NEWLIB). Compiled with -fno-lto always.
3. crates/fbuild-build/src/shrink/newlib_stdio_manifest.rs — per-newlib-version map of which symbols vfprintf.o / svfprintf.o co-export, used to size the complete-shadow archive. CI test diffs xtensa-esp-elf-nm output against the manifest and fails on drift.
4. crates/fbuild-build/src/shrink/printf_thin_archive.rs — combines the picolibc sources + a thin vfprintf / _vfprintf_r / printf / _printf_r / __sbprintf / _svfprintf_r / vsnprintf etc. shim into libprintf_thin.a. Each entry point sets up a stack-allocated cookie FILE whose lock is pre-bound to __lock___libc_recursive_mutex (ESPHome fix for the lazy-mutex leak), then forwards to picolibc's __d_vfprintf. Guarded by #ifdef __PICOLIBC__ #error #endif so the archive refuses to build under picolibc-as-libc.
5. crates/fbuild-build/src/shrink/spec_emitter.rs — generates printf-thin.specs for the resolved toolchain; cached per-toolchain under ~/.fbuild/cache/printf-thin/<toolchain-hash>/.
6. crates/fbuild-build/src/shrink/wrap_fallback.rs — -Wl,--wrap= flag set + noinline,externally_visible stub TU, used when the auto-resolver picks printf mode (single-knob debugging) or when spec-file emission fails.
7. crates/fbuild-build/src/shrink/applier.rs — translates the chosen mode into compile flags, link flags (--specs= primary, --wrap fallback), extra TUs / archives, and sdkconfig overrides (aggressive only). Auto-disables when --platformio is set (byte-identical PIO output, per review comment). Auto-disables when CONFIG_LIBC_NEWLIB_NANO_FORMAT=y is detected in resolved sdkconfig (avoids net-negative outcome, per review comment).
8. crates/fbuild-build/src/shrink/probe.rs — fail-closed libc probe via a 5-line probe TU with #ifdef __PICOLIBC__\n#error PICOLIBC\n#endif, inspecting preprocessor exit code. On any probe failure → default to off, log warning (per review comment).
9. crates/fbuild-build/src/build_info.rs — record shrink: { mode, resolved, applied: [...], strategy: "spec-file"|"wrap" } in build_info.json so fbuild bloat reports can label results.
10. crates/fbuild-build/src/compile_many.rs:121 — extend seed_stage2_core_from_stage1 to include printf_thin.a and the spec file so multi-sketch CI builds amortize the printf-thin work in stage 1 (per review comment).
11. Tests: an esp32s3 build under each mode (off, safe, aggressive, printf, auto), assert the symbol-table delta (_vfprintf_r absent under safe+ etc.) — small integration test that runs in CI. Additional test: a String-heavy sketch that exercises dtostrf to confirm _dtoa_r is still present and float formatting works (per review comment, dtostrf is a separate _dtoa_r caller and won't disappear from the shrink).

Decisions

• Default is --shrink=auto (confirmed). Passing no flag is equivalent to --shrink=auto. The auto-resolver is conservative (picks off whenever the libc is already optimal), so on platforms with no auto-shrink the behavior is byte-for-byte identical to today. On Arduino-ESP32 IDF 5.x the printf-thin wrap fires and saves ~24 KB. A BREAKING_CHANGES.md line should call this out.

Open design questions

1. Composability for aggressive — do we want --shrink=aggressive --shrink-skip=coredump from the start? Proposal: no for v1. Five modes + platformio.ini's fbuild_shrink_skip covers the long tail. Composition is non-breaking to add later.
2. aggressive failure mode when sdkconfig cannot be regenerated — loud failure with hint to use --clean or downgrade to --shrink=safe, vs silent degrade. Proposal: loud failure — silent feature loss is worse than an explicit error.
3. Telemetry in build_info.json — record shrink: { mode, resolved, applied } so fbuild bloat and CI bloat-budget gates can label reports. Proposal: yes — costs nothing, makes the per-symbol report self-describing.
4. Single --shrink on the global Cli vs per-subcommand — fbuild already duplicates --clean, --environment, --verbose between the global Cli and per-subcommand Build/Deploy. Match that pattern.

Risks and caveats

• Newlib symbol-list drift (spec-file primary path): libprintf_thin.a shadows newlib's vfprintf.o/svfprintf.o co-exports. If newlib reshuffles those objects, our shadow misses a symbol and the linker pulls newlib's full object back in. Mitigation: per-newlib-version symbol manifest + CI test that diffs nm against the manifest. Newlib's stdio layout has been stable for ~15 years; the manifest changes once every few years.
• LTO and --wrap fallback: when --shrink=printf selects the wrap path (debugging mode), LTO can inline wrapped symbols across the wrap boundary if our TUs are in the same LTO unit as the sketch. Mitigation is two-step: vendored picolibc compiled with -fno-lto, and stub functions marked __attribute__((noinline, externally_visible)). Linker scripts cannot fix this because they run after LTO inlining.
• Bringing _dtoa_r back via strtod: ~3.5 KB of the savings only holds while nothing in the sketch calls strtod/atof/Arduino::String::toFloat. Once a sketch parses floats from strings, the dtoa block resurrects. Fix is to also bring picolibc's strtod into the shadow archive (~2 KB cost) — track separately if it becomes a real regression. The bundled scanf wrap (v1 per review comment) brings picolibc's strtod along automatically.
• Arduino::String += float keeps _dtoa_r alive: Arduino-ESP32's dtostrf calls _dtoa_r directly (not via vfprintf), so it stays linked after --shrink=safe. No regression in behavior, no extra savings beyond the printf core. CI test on a String-heavy sketch (FastLED Apa102 example) confirms this is correct.
• _REENT semantics under FreeRTOS: picolibc tinystdio calls write(fd, ...) directly, which in IDF resolves to _write_r(_REENT, ...) via the VFS layer. Errno is set correctly because _write_r writes through _REENT. Stub must not read/write errno between the picolibc call and return; CI test confirms snprintf(NULL, 0, ...) does not clobber errno. The __lock___libc_recursive_mutex pre-binding fix (from ESPHome's pre-drop work on the picolibc cookie path) handles flockfile's lazy heap-allocated mutex.
• Float quality: picolibc tinystdio with io-float-exact=false (dtoa-engine, not Ryu) is slightly less accurate than newlib's _dtoa_r on edge cases (off-by-1-ulp on a tiny fraction of values). For embedded logging this is fine; for code that depends on exact IEEE round-trip from snprintf -> strtod, this is not safe. Document under --shrink=printf/--shrink=safe and add BREAKING_CHANGES.md line.
• Banker's vs half-up rounding: picolibc dtoa-engine uses round-half-up; newlib _dtoa_r uses banker's rounding (round-half-to-even). Sketches with golden-output tests will diverge. Recommend --no-shrink for byte-exact reproduction.
• IRAM placement on ESP32: the spec-file shadow archive lives in flash (matches stock newlib placement). ISR / panic-handler paths in IDF already use esp_rom_printf or esp_log_early_printf, not vprintf. CI test: trigger esp_panic, confirm shadow not reached.
• Forward compat is real: on ESP-IDF 6.0+ the compiler defines __PICOLIBC__, the shadow archive's #error guard fires, the spec file becomes inert (no newlib vfprintf symbol to shadow), and the firmware gets picolibc natively. Same forward-compat mechanism as ESPHome.

Related

• fbuild bloat: emit and preserve firmware.map automatically #491 — fbuild bloat: emit and preserve firmware.map automatically. The build-output reporting above depends on having archive-attributed symbols, which depends on the map.
• Investigation thread that produced these numbers (June 2026 session): printf-family bloat on esp32s3, picolibc tinystdio extraction cost, ESPHome --wrap precedent.

Acceptance

• fbuild build --shrink and fbuild build --no-shrink both work, --shrink=auto|off|safe|aggressive|printf parses.
• Auto-resolver picks safe for Arduino-ESP32 3.3.7 on ESP32-S3 and prints the decision via the green one-liner.
• Default mode (--shrink=safe) uses the spec-file path (printf-thin.specs + libprintf_thin.a shadow archive); --shrink=printf uses the --wrap fallback for single-knob debugging.
• A clean build with --shrink=safe produces a firmware where xtensa-esp-elf-nm shows no _vfprintf_r or _svfprintf_r, and _dtoa_r is absent if the sketch does not call strtod / dtostrf.
• Flash size on tests/platform/esp32s3 -e esp32s3 drops by >= 18 KB (conservative target; measured ceiling is ~24 KB).
• build_info.json records the resolved mode, applied appliers, and link strategy ("spec-file" or "wrap").
• --no-shrink produces a byte-for-byte identical (or modulo timestamp) ELF to today's build.
• CI test diffs newlib's nm output against newlib_stdio_manifest.rs; fails on drift so the shadow archive stays complete across toolchain bumps.