Skip to content

fix(esp32): prevent ESP32-S3 serial reset deadlock and boot-mode lockup #532

New issue
New issue
Closed
Closed
fix(esp32): prevent ESP32-S3 serial reset deadlock and boot-mode lockup#532

Description

@zackees
zackees
opened on Jun 9, 2026

Context

During FastLED AutoResearch testing on an attached ESP32-S3, the board could be left in a temporarily bricked/dead state after the upload/monitor/RPC handoff path. The failure looked like a host-side deadlock plus a board that stopped responding to RPC.

Observed on Windows with the ESP32-S3 USB serial/JTAG port:

  • S3 upload/reset used COM22 successfully.
  • After monitor/RPC handoff attempts, the board could stop responding to JSON-RPC even though the serial device still existed.
  • A bad serial-line state could put the S3 into ROM download mode:
    • boot:0x23 DOWNLOAD(USB/UART0)
    • waiting for download
  • Recovery required an explicit DTR/RTS reset sequence. This is easy to miss if fbuild or a follow-on pyserial client is holding the port or has left DTR/RTS asserted in a bootloader-selecting state.
  • This is adjacent to, but distinct from, fix(monitor): release Windows COM port after monitor exits #531 where fbuild monitor --timeout ... can leave the Windows COM port owned by the daemon until fbuild daemon stop is run.

The practical result is that a test runner can look deadlocked while the S3 is effectively stuck in the boot ROM or otherwise unavailable for RPC.

Proposal

Harden the ESP32 serial reset/monitor/RPC handoff path so fbuild never leaves ESP32-S3 boards in an unrecoverable or confusing state:

  • Centralize ESP32 DTR/RTS handling for reset and boot mode selection.
  • Ensure normal monitor/RPC handoff leaves DTR/RTS in the run-firmware state, not the download-mode state.
  • Detect common ESP ROM download-mode output such as waiting for download after a reset/monitor operation and either recover automatically or fail with a targeted diagnostic.
  • Add watchdog-style timeout handling around monitor/serial operations so a board that stopped responding does not leave fbuild appearing deadlocked.
  • Always release the COM port before handing it to a pyserial/RPC client, including failure paths.

Acceptance criteria

  • Repeated deploy -> monitor --timeout -> pyserial/RPC open cycles on an ESP32-S3 do not leave the board in ROM download mode.
  • If the board emits waiting for download, fbuild reports that exact boot-mode problem or resets it back to application mode.
  • A failed or timed-out monitor/RPC operation does not require fbuild daemon stop before another process can recover the board.
  • The fbuild logs show enough DTR/RTS/reset context to diagnose future S3 boot-mode lockups.
  • The behavior is covered by at least a regression test or scripted repro path for the ESP32-S3 serial reset state machine.

Related issues

Activity

  1. fastled-project-sync
    added this to FastLED Trackeron Jun 9, 2026
  2. fastled-project-sync
    moved this to Triage in FastLED Trackeron Jun 10, 2026
  3. zackees
    added 2 commits that reference this issue on Jun 13, 2026

    fix(esp32): detect ESP ROM download-mode in serial monitor (#532) (#547)

    44adabf

    feat(serial): add esp_reset hard-reset DTR/RTS primitive (#532) (#577)

    abc7ab6

  4. zackees commented on Jun 14, 2026

    @zackees
    zackees
    on Jun 14, 2026
    MemberAuthor

    Status — what's shipped, what's left

    Three PRs have landed against this issue; tracking what each one moved:

    PR Landed Covers
    #547 2026-06-13 12:20 UTC Adds `fbuild_serial::boot_mode::detect_download_mode` — case-insensitive substring match against `waiting for download` / `boot:0x.. DOWNLOAD(USB/UART)`. Wires it into `MonitorState::process_line` as a `tracing::warn!`
    #557 2026-06-13 15:45 UTC Promotes the detection from "warn and keep waiting" to `MonitorOutcome::Error` with the actionable diagnostic. Halts on the first download-mode signal so the monitor doesn't burn its full timeout against a bricked board. `#531`'s port-release path runs on the outcome-error, so the COM port is free for an esptool / pyserial recovery client immediately afterward
    #577 2026-06-14 13:20 UTC Adds `fbuild_serial::esp_reset::hard_reset_blocking` — the matching DTR/RTS recovery primitive (DTR=low → RTS=high pulse → RTS=low, 100 ms hold). Trait-based (`DtrRtsControl` with a blanket impl over `serialport::SerialPort`) so 4 hardware-free unit tests lock down the canonical sequence, the pulse duration, error propagation, and the DTR-failure short-circuit invariant

    Acceptance criteria audit

    • "Board emits `waiting for download` → fbuild reports that exact boot-mode problem" — done (fix(esp32): detect ESP ROM download-mode in serial monitor (#532) #547 + fix(esp32): surface ESP ROM download-mode lockup instead of silent timeout #557).
    • "A failed or timed-out monitor/RPC operation does not require `fbuild daemon stop` before another process can recover the board" — done (fix(esp32): surface ESP ROM download-mode lockup instead of silent timeout #557, via the existing port-release path).
    • "Behavior covered by at least a regression test or scripted repro path for the ESP32-S3 serial reset state machine" — done (5 detection tests in `boot_mode.rs`, 4 reset-sequence tests in `esp_reset.rs`).
    • [~] "Repeated `deploy → monitor --timeout → pyserial open` cycles do not leave the board in ROM download mode" — primitive shipped, not wired: the monitor currently fast-fails with `MonitorOutcome::Error` and leaves the recovery to the caller. The next PR wires `hard_reset_blocking` into the monitor's download-mode branch so the daemon attempts one automatic recovery before returning the error. Opt-in (per-request flag) so existing fast-fail behaviour stays the default until hardware verification.
    • [~] "The fbuild logs show enough DTR/RTS/reset context to diagnose future S3 boot-mode lockups" — partial: `esp_reset::hard_reset_blocking` logs every DTR/RTS transition at `tracing::debug!` plus a completion line at `tracing::info!`. The existing open-time DTR/RTS assertions in `fbuild_serial::manager` (`write_data_terminal_ready(true) / write_request_to_send(true)`) still only emit on the failure path. Trivial follow-up: add a `tracing::debug!` covering the success path so a complete log scan reconstructs the line-control history.

    Proposal items audit (from issue body)

    • Centralize ESP32 DTR/RTS handling for reset — done via `esp_reset`.
    • Centralize ESP32 DTR/RTS handling for boot mode selection — the open-time `set_dtr(true)/set_rts(true)` pair in `manager.rs` is generic ABI hygiene, not ESP boot-mode selection. Currently no module knows whether the port is an ESP, so it can't pick a boot-mode-safe default. Tracked separately if needed.
    • Ensure normal handoff leaves DTR/RTS in run-firmware state, not download-mode state — defensible today because the open-time pair sets both lines HIGH, and the ESP wiring (RTS HIGH = EN HIGH = release reset; DTR HIGH would assert BOOT but only matters during a reset edge). The auto-recovery PR will explicitly end the sequence with DTR=low, RTS=low so the chip is left in the cleanest possible state.
    • Detect ESP ROM download-mode output and either recover automatically or fail with a targeted diagnostic — fail path done; auto-recover path = the next PR.
    • Add watchdog-style timeout handling around monitor/serial operations — already covered by the existing `timeout_secs` machinery (`MonitorState::timed_out`); fix(esp32): surface ESP ROM download-mode lockup instead of silent timeout #557's fast-fail closed the practical "silent timeout" failure mode.
    • Always release the COM port before handing it to a pyserial/RPC client, including failure paths — done via fix(esp32): surface ESP ROM download-mode lockup instead of silent timeout #557 + the fix(monitor): release Windows COM port after monitor exits #531 port-release wire.

    Concrete remaining work for the next PR

    1. New optional field on `MonitorRequest` (and the in-process equivalent): `auto_recover_from_download_mode: bool` (default `false` to preserve current behaviour).
    2. New `MonitorOutcome::RecoverDownloadMode { signal: BootModeSignal }` variant returned by `MonitorState::process_line` only when the flag is set.
    3. In `monitor()` HTTP handler (which already holds `ctx.serial_manager`), match that outcome → borrow the port → invoke `esp_reset::hard_reset_blocking` inside `tokio::task::spawn_blocking` → re-attach the monitor with the same remaining timeout. If recovery fails or the chip immediately re-emits a download-mode signal, return `MonitorOutcome::Error` (the existing pre-feat(serial): add esp_reset hard-reset DTR/RTS primitive (#532) #577 behaviour).
    4. Add the missing success-path `tracing::debug!` entries to `manager.rs` for the line-control history.

    Hardware verification is a separate beat (the issue's last acceptance line); the unit tests in `esp_reset.rs` already prove the sequence is correct against a mocked port. End-to-end recovery on a real ESP32-S3 stays a follow-up beat until someone with the board can run the regression.

  5. zackees
    added 2 commits that reference this issue on Jun 14, 2026

    fix(#580): keep deploy.rs under LOC gate + resolve intra-doc links

    43f3f72

    feat(daemon): opt-in ESP auto-recovery from ROM download mode (#532) (#…

    fb2866a

  6. zackees commented on Jun 14, 2026

    @zackees
    zackees
    on Jun 14, 2026
    MemberAuthor

    Closing — all acceptance criteria met via the following PR chain on main:

    # Landed Scope
    #547 2026-06-13 12:20 UTC fbuild_serial::boot_mode::detect_download_mode — case-insensitive substring match against waiting for download / boot:0x.. DOWNLOAD(USB/UART). Wires it into MonitorState::process_line as a tracing::warn!
    #557 2026-06-13 15:45 UTC Promotes detection to MonitorOutcome::Error with the actionable diagnostic. Halts on the first download-mode signal so the monitor doesn't burn its full timeout against a bricked board. The #531 port-release path runs on the outcome-error, so the COM port is free for an esptool/pyserial recovery client immediately afterward
    #577 2026-06-14 13:20 UTC fbuild_serial::esp_reset::hard_reset_blocking — DTR/RTS recovery primitive (DTR=low → RTS=high pulse → RTS=low, 100 ms hold). Trait-based DtrRtsControl with a blanket impl over serialport::SerialPort so 4 unit tests lock down the canonical sequence, pulse duration, error propagation, and the DTR-failure short-circuit invariant — all hardware-free
    #580 2026-06-14 17:47 UTC Opt-in auto-recovery wire-up: MonitorRequest.auto_recover_from_download_mode flag (default false), new MonitorOutcome::RecoverDownloadMode { signal } variant, SharedSerialManager::esp_hard_reset borrowing the owned port, fold-before-match in the HTTP monitor handler, success-path tracing::debug! for the open-time DTR/RTS assertions in manager.rs

    Acceptance criteria — final audit

    Follow-up beats (separate tracking)

    • End-to-end hardware verification on a real ESP32-S3. The unit tests prove the sequence and the wire-up are correct against a mocked port; live-hardware soak (repeated deploy → monitor → recovery cycles on a physical board) stays a hardware-verification beat for whoever has the board, not a code blocker. Anyone running the AutoResearch harness on an S3 can flip auto_recover_from_download_mode: true on their monitor RPC call to exercise the new path.
    • Wiring the same flag into the post-deploy monitor in deploy.rs would extend recovery to the fbuild deploy --to monitor flow. Currently passes false (fast-fail). Worth a focused follow-up PR once the standalone-monitor path has hardware miles.
  7. zackees
    closed this as completedon Jun 14, 2026
  8. zackees
    added 6 commits that reference this issue on Jun 14, 2026

    fix(ci): split script_runtime tests under LOC gate + format fixes (#581…

    b823b7d

    fix(ci): split script_runtime tests under LOC gate + format fixes (#581…

    3d17cb7

    feat(build): opt-in lite-SCons extra_scripts harness (#553) (#581)

    34293b0

    chore(deps): bump fbuild 2.2.27 -> 2.2.28 (PyPI 2026-06-15) (#3056)

    79ab19e

    docs(serial): USB CDC DTR/RTS control-line semantics matrix (#689) (#701

    6cfc680

    feat(serial): per-family BootModeClassifier registry + ArmHardFault m…

    758e6e1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions