Skip to content

test: validate per-chip flash offsets against an authoritative source (recurring ROM-offset bug) #279

New issue
New issue
Closed
Closed
test: validate per-chip flash offsets against an authoritative source (recurring ROM-offset bug)#279

Description

@zackees
zackees
opened on May 29, 2026

Context

Per-chip flash layout values — most critically the second-stage bootloader offset — are hand-maintained in the per-MCU JSON configs (crates/fbuild-build/src/esp32/configs/*.json → esptool.flash_offsets.bootloader). These are ROM-defined constants: get one wrong and the chip's ROM reads garbage at its fixed load address and enters an invalid header: 0x… reboot loop. Nothing currently cross-checks these values against an authoritative source, so a wrong value ships silently and only surfaces as a dead board.

This has bitten us repeatedly. Most recently #278: esp32p4 and esp32c5 both shipped bootloader: "0x0" but the ROM expects 0x2000, producing the invalid header: 0x47550ff7 boot loop on ESP32-P4 hardware. The pattern (a new/edited chip config carrying the wrong offset) is the recurring failure mode, not a one-off.

Authoritative values (ESP-IDF ESP_BOOTLOADER_OFFSET, also exposed as build.bootloader_addr in arduino-esp32 boards.txt):

Offset Chips
0x0 esp32s3, esp32c2, esp32c3, esp32c6, esp32h2, esp32c61
0x1000 esp32, esp32s2
0x2000 esp32p4, esp32c5

Current state in-tree after #278 (all correct now, but only guarded by a hand-written table):

esp32=0x1000  esp32s2=0x1000
esp32s3=0x0   esp32c2=0x0  esp32c3=0x0  esp32c6=0x0  esp32h2=0x0
esp32c5=0x2000  esp32p4=0x2000

#278 added test_bootloader_flash_offsets_match_rom (crates/fbuild-build/src/esp32/mcu_config.rs), but it asserts each config against a hardcoded table in the same repo — it catches an accidental edit to an existing config, yet a newly-added chip can still ship a wrong offset by being added to both the config and the table with the same mistake. We have board-validation tooling already (ci/validate_boards.py, ci/board_sources.py) that pulls external sources, which is the natural place to cross-check.

Proposal

Validate the flash offsets against an authoritative external source rather than (only) an in-repo table:

  • Extend ci/validate_boards.py (or add a sibling check) to read build.bootloader_addr from the installed arduino-esp32 boards.txt (and/or ESP-IDF ESP_BOOTLOADER_OFFSET) and assert it matches each esp32*.json's esptool.flash_offsets.bootloader. Run it in CI.
  • Treat a missing/mismatched offset for any supported chip as a hard failure, so adding a new chip forces the correct value.
  • Consider extending the same authoritative cross-check to the other flash_offsets (partitions 0x8000, firmware 0x10000) and to non-ESP32 platforms that carry analogous ROM-defined layout constants.
  • Keep the fast in-repo unit test as a first line of defense, but make the CI/external check the source of truth.

Acceptance criteria

  • A CI-run check fails when any esp32*.json bootloader offset disagrees with the authoritative source (arduino-esp32 boards.txt build.bootloader_addr and/or ESP-IDF), verified by temporarily mutating one config and observing the failure.
  • The check covers every chip in crates/fbuild-build/src/esp32/configs/ and fails (not silently skips) on an unknown/missing chip.
  • Adding a new ESP32 variant with a wrong/absent bootloader offset cannot pass CI.
  • Docs/comment point maintainers at the authoritative source so the value isn't guessed.

Decisions

  • Type: test/CI enhancement (not a code bug) — the offsets are currently correct in-tree (fixed in ESP32-P4 (eco2) won't boot: bootloader flashed at 0x0 (should be 0x2000) + eco5-linked bootloader/app crash on eco2 silicon #278); this issue is the guardrail to stop the class of bug recurring.
  • Priority: P2 — no active breakage right now, but it's a repeat foot-gun with severe (dead-board) impact; worth doing before the next chip is added.
  • Authoritative source: arduino-esp32 boards.txt build.bootloader_addr as the primary cross-check, since fbuild already resolves that framework and ci/board_sources.py knows how to read it; ESP-IDF ESP_BOOTLOADER_OFFSET as a secondary reference. Triage may prefer one over the other.
  • Scope: bootloader offset first, other flash_offsets/platforms as follow-up — bootloader offset is the one with catastrophic, ROM-level failure; the rest are lower-risk.
  • Home: extend ci/validate_boards.py as the guessed location; can be re-routed.

Related issues

Activity

  1. zackees commented on May 29, 2026

    @zackees
    zackees
    on May 29, 2026
    MemberAuthor

    Parallel audit: which boards actually have this problem

    Ran three parallel sub-agent audits (ESP32 config cross-check vs authoritative arduino-esp32 boards.txt; board-DB MCU coverage; all non-ESP32 platforms). Results below; concrete fixes as a sub-issue checklist at the bottom.

    Severity note: "won't boot" here means the invalid header ROM reboot-loop from #278 — the device is recoverable by reflashing (the first-stage bootloader is in mask ROM and can always re-enter download mode). It is not a permanent hardware brick. The impact is that the board appears dead out of the box and automated flows (e.g. autoresearch) hang on it until a correct image is flashed.

    Findings

    1. All 9 existing ESP32 configs are correct. Every esp32*.json bootloader/partitions/firmware offset matches the chip's authoritative *.build.bootloader_addr in boards.txt (esp32/s2=0x1000; s3/c2/c3/c6/h2=0x0; c5/p4=0x2000; partitions=0x8000, firmware=0x10000). After #278 there is no currently-wrong offset in the shipped configs.

    2. Coverage gap — esp32c61 has no config. boards.txt defines 10 ESP32 families; fbuild ships 9. The board esp32-c61-devkitc1-n8r2.json (build.mcu: esp32c61, platform: espressif32) maps to a chip with no esp32c61.json. Authoritative offset for c61 per boards.txt:1017 is 0x0 (note: this corrects the 0x2000 guess in the issue body's table — c61 is 0x0).

    3. The real amplifier — deploy silently falls back to esp32's 0x1000. get_mcu_config correctly hard-errors on an unknown MCU, and the build path propagates it (orchestrator/build.rs:64 uses ?). But the deploy path swallows it:

    // crates/fbuild-daemon/src/handlers/operations/deploy.rs:380-383
let mcu_config = get_mcu_config(&board_config.mcu)
    .unwrap_or_else(|_| get_mcu_config("esp32").unwrap());  // <-- silent wrong 0x1000

    So any uncovered/new ESP32 variant doesn't error — it flashes the bootloader at esp32's 0x1000, which makes every RISC-V variant (0x0) and C5/P4 (0x2000) fail to boot (invalid header reboot loop, recoverable by reflashing). This unwrap_or_else(esp32) is exactly how a coverage gap produces a non-booting board out of the box, and is the highest-value fix.

    4. No non-ESP32 platform is affected. ESP8266 carries no fbuild-maintained offset (app flashes at 0x0 via esptool defaults). RP2040's 0x10000000 is the XIP flash base (silicon-fixed, UF2-deployed), not an upload offset. AVR/STM32/Teensy/nRF52/SAM/SAMD/CH32V/Renesas/SiLabs/Apollo3/generic-ARM use avrdude/UF2/dfu/loader with no ROM bootloader offset. The hand-maintained-offset risk is unique to ESP32.

    Blast radius (espressif32 boards, 297 total)

    esp32=141, esp32s3=88, esp32s2=28, esp32c3=18, esp32c6=11, esp32p4=4, esp32c5=4, esp32h2=1, esp32c2=1, esp32c61=1 (no config).

    Sub-issues to fix

    • Remove the silent unwrap_or_else(get_mcu_config("esp32")) fallback in deploy.rs:380-383 so deploy fails loudly on an unknown MCU (like the build path), instead of flashing a wrong 0x1000 bootloader. Highest priority — this is what turns every coverage gap into a non-booting (until reflashed) board.
    • Add an esp32c61.json config (bootloader 0x0, partitions 0x8000, firmware 0x10000; verify against ROM/boards.txt:1017) so esp32-c61-devkitc1-n8r2 is covered.
    • Add the CI/authoritative validation check described in the issue body (assert every esp32*.json offset == boards.txt build.bootloader_addr; hard-fail on a missing chip) so a new variant can't ship a wrong/absent offset.
    • Fix the issue-body table: esp32c61 authoritative bootloader offset is 0x0, not 0x2000.

    No sub-issues needed for non-ESP32 platforms.

  2. zackees
    added a commit that references this issue on May 29, 2026

    Merge pull request #281 from FastLED/fix/esp32-p4-eco2-and-flash-offs…

    5cb265a

  3. zackees commented on May 30, 2026

    @zackees
    zackees
    on May 30, 2026
    MemberAuthor

    The guardrail proposed here landed on main with ci/check_flash_offsets.py (added via 3758098e) and is wired into .github/workflows/validate-boards.yml as the Validate ESP32 flash offsets match authoritative source step.

    Just verified locally on main:

    $ uv run python ci/check_flash_offsets.py --download
Authoritative source: pioarduino/arduino-esp32 release tag 3.3.7 (downloaded)
platform.txt default build.bootloader_addr = 0x1000
configs directory: …/crates/fbuild-build/src/esp32/configs

CHIP         CONFIG     AUTHORITATIVE  PART     FW         RESULT
-----------------------------------------------------------------
esp32        0x1000     0x1000         0x8000   0x10000    PASS
esp32c2      0x0        0x0            0x8000   0x10000    PASS
esp32c3      0x0        0x0            0x8000   0x10000    PASS
esp32c5      0x2000     0x2000         0x8000   0x10000    PASS
esp32c6      0x0        0x0            0x8000   0x10000    PASS
esp32h2      0x0        0x0            0x8000   0x10000    PASS
esp32p4      0x2000     0x2000         0x8000   0x10000    PASS
esp32s2      0x1000     0x1000         0x8000   0x10000    PASS
esp32s3      0x0        0x0            0x8000   0x10000    PASS

All 9 esp32 config(s) match the authoritative flash offsets.

    Acceptance criteria covered:

    • CI check fails when any esp32*.json bootloader offset disagrees with the authoritative source (arduino-esp32 boards.txt build.bootloader_addr, with platform.txt default applied for chips that don't override). The check also fails on unknown/missing chips and on partitions != 0x8000 / firmware != 0x10000.
    • Covers every chip currently in crates/fbuild-build/src/esp32/configs/.
    • A new ESP32 variant with a wrong/absent bootloader offset cannot pass CI.
    • Authoritative source is documented inside ci/check_flash_offsets.py and pointed at via --download <tag>.

    Closing — the recurring class of bug behind #278 now has a CI guardrail.

  4. zackees
    closed this as completedon May 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions