Skip to content

investigate(rp2040): readable BOOTSEL volume stalls on first write; make watchdog cleanup handle-safe #1161

Open

Description

@zackees

Context

Two RP2040 boards have now refused programming. The current failure has a specific, repeatable state:

  • The RP2040 enters BOOTSEL and enumerates as USB\VID_2E8A&PID_0003\E0C9125B0D9B.
  • Its mass-storage interface (MI_00) is healthy.
  • Windows mounts RPI-RP2 on G: and reports the volume healthy.
  • INFO_UF2.TXT and other reads succeed.
  • Writes do not complete:
    • fbuild's NEW.UF2 write stalls;
    • a normal Windows file copy stalls;
    • a paced write-through test using 8 KiB chunks stalled before its first completed chunk and recorded zero progress.
  • Windows logged Disk events 51 and 153 during the attempts, including retries at LBAs 0x134 and 0x14c.
  • Restart Manager reported no user-mode holder, and an exclusive volume open succeeded between attempts.

The last point does not prove that the failure is device-side. Kernel storage filters, filesystem behavior, USB transport faults, QSPI programming behavior, and a host/controller problem can all exist without a user-mode volume handle. A USB trace is required before claiming that the RP2040 bulk-OUT endpoint failed to acknowledge the transfer.

The separate MI_01 PICOBOOT interface currently has no Windows driver. That does not explain why the healthy MI_00 mass-storage interface can be read but not written.

fbuild regression question

RP2040 deploy hardening was merged in commit debc9963 through PR #1083.

The relevant commit boundary is:

  1. d5144e0111d04b390f431e9700211a65cfee2f9d — immediately before the hardening
  2. a3d8c58b94111a4e055f66d67d4ff8ad702fd730 — fresh-enumeration retries
  3. c472f56f5989a4507321e862fdb62d7c1bcf309f — watchdog and broader hardening
  4. debc9963527e0a5a09f1224883b02d63ba174425 — merged hardening
  5. 16bb9ffae6aeab38957176357425110b002d98b1 — current main during this reproduction

Static comparison shows that the low-level first-write shape was not materially changed by #1083: both sides open/truncate NEW.UF2 and make a whole-buffer write_all call.

The hardening did introduce a definite recovery hazard. run_with_watchdog deliberately abandons a timed-out worker thread:

/// Run `work` on a dedicated thread and give up after `budget`. A storport
/// retry storm behind a sick hub can block the NEW.UF2 `write_all` for
/// minutes with no output; this turns that hang into an actionable failure
/// that feeds the fresh-enumeration retry gate. On timeout the worker thread
/// is deliberately abandoned, not joined: it only touches its own locals and
/// its channel send fails silently once this receiver is dropped, so the
/// wedged kernel write can unblock (or not) without holding up the deploy.
fn run_with_watchdog<T: Send + 'static>(
budget: Duration,
label: &str,
work: impl FnOnce() -> Result<T> + Send + 'static,
) -> Result<T> {
let (sender, receiver) = std::sync::mpsc::channel();
std::thread::spawn(move || {
let _ = sender.send(work());
});
match receiver.recv_timeout(budget) {

That worker owns the open NEW.UF2 file. If the kernel write remains blocked, the thread and handle can remain alive inside the daemon after fbuild reports a timeout. This can contaminate retries and later deploy attempts even if the watchdog did not cause the original first-write stall.

The existing watchdog test verifies that the caller returns quickly by leaving a five-second worker running. It does not verify worker termination, handle release, or whether an immediate second write can open the destination.

There is also a local version-isolation risk:

  • Global executable: C:\tools\python13\Scripts\fbuild.exe, reporting 2.5.2
  • FastLED virtual-environment executable: reporting 2.5.3
  • Installed Python metadata in that environment: reporting 2.5.2

Every hardware result therefore needs to record the exact executable, executable hash, source commit, daemon port, and daemon binary used.

Pinned artifact evidence

The current artifact is:

  • Size: 882,688 bytes
  • SHA-256: 1C770999678BB7563A81D36C696DC44C074A712E25F0DE68A8A83F859E8AF0E0
  • 1,724 UF2 blocks of 512 bytes
  • RP2040 family ID: 0xE48BFF56
  • Target range: 0x10000000 through 0x1006BB00
  • Block magic, block numbering, payload lengths, family ID, and declared block count are structurally valid

An earlier deploy transcript reported a different artifact size. Commit comparisons must therefore use one pinned artifact without rebuilding between runs.

Proposal

1. Produce a deterministic reproduction record

Before every attempt, record:

  • physical board serial and VID:PID;
  • PnP health for both BOOTSEL interfaces;
  • drive letter and USB hub/root-port topology;
  • exact fbuild executable path, version, hash, source commit, and daemon binary;
  • a unique five-digit FBUILD_DAEMON_PORT;
  • artifact path, size, and SHA-256;
  • relevant Windows Disk 51/153 events before and after the attempt;
  • whether any fbuild worker/process/volume handle remains after timeout.

Use an outer killable process boundary for any experiment that may block. Do not allow one timed-out attempt to contaminate the next case.

2. Run a transport/artifact matrix

On the same board, cable, direct motherboard port, and clean BOOTSEL enumeration, compare:

  1. the previously proven RAM-only UF2;
  2. a small official known-good RP2040 flash Blink UF2;
  3. the pinned fbuild-generated flash UF2.

Interpretation:

  • RAM succeeds but both flash images fail: investigate the flash-target/QSPI path.
  • Official flash succeeds but fbuild flash fails: investigate artifact generation or layout.
  • All three fail on the first storage request: investigate the Windows/USB/volume path.
  • The result changes with fbuild commit while the pinned artifact and physical setup remain identical: investigate an fbuild deploy regression.

Also repeat the decisive case using:

  • a known-good data cable;
  • a direct USB 2.0-capable motherboard port;
  • a second clean host, preferably Linux.

On Linux, a guarded comparison between a normal mounted-volume copy and an unmounted raw block-device write may distinguish filesystem integration from USB/device behavior. The exact block device must be verified before any raw write.

3. Capture the failing layer

Capture one failure with:

  • USBPcap/Wireshark, to determine whether the first mass-storage bulk-OUT transaction leaves the host and what response/reset sequence follows;
  • Process Monitor filtered to the selected volume and fbuild process;
  • Windows Disk, StorPort, Kernel-PnP, and USB events;
  • read-only fltmc/PnP inspection when elevated access is available.

The trace should classify the failure without assuming that readable media plus a missing user-mode handle proves a board fault.

4. Perform an exact hardening A/B

Build and run isolated versions at the commit boundary listed above:

  • use separate worktrees/installations;
  • use separate daemon ports and state directories;
  • verify both CLI and daemon provenance;
  • use the same pinned UF2;
  • start with the pre-hardening build after a clean reboot;
  • do not run an older build after a watchdog failure unless every newer worker/process/handle has been proven gone.

Decision criteria:

  • d5144e01 repeatedly succeeds while c472f56f fails in the clean identical setup: hardening regression.
  • Both fail on the same first storage operation: the initial stall predates or is independent of the hardening.
  • Both fail initially, but only watchdog-enabled builds retain a worker/handle or poison the next attempt: underlying transport failure plus a separate fbuild recovery regression.

5. Make timeout cleanup handle-safe

If the watchdog path is retained, it must not abandon a thread that owns a removable-volume file handle.

Use either:

  • a killable helper process whose termination closes the handle; or
  • cancellable Windows overlapped I/O using CancelIoEx, followed by bounded completion and thread joining.

After timeout:

  • the worker must be terminated or joined;
  • the NEW.UF2 handle must be closed;
  • stale volume paths must be discarded;
  • retries may occur only after a verified fresh BOOTSEL enumeration;
  • an immediate second open/write probe must not fail because of the prior attempt.

Add a deterministic regression test covering timeout followed by immediate handle reacquisition and a second write.

6. Use PICOBOOT only as a controlled comparison

PICOBOOT/picotool can be used as a transport control on another host, or where the interface already has a compatible driver. Do not install or rebind drivers on the current machine as part of this issue.

Acceptance criteria

  • A reproducible log captures exact board, artifact, executable, daemon, USB topology, and Windows event identities.
  • The RAM-only, official flash, and fbuild flash artifact matrix is completed without rebuilding between cases.
  • At least one USB trace identifies whether the first write leaves the host and what fails next.
  • The pre-hardening through current-main A/B identifies whether harden(rp2040): hub-path deploy robustness — retries, picotool fallback, topology diagnostics, tunable budgets #1083 changed the initial failure.
  • The initial transport failure and any post-timeout contamination are reported as separate findings.
  • A watchdog timeout leaves no live worker, helper, or NEW.UF2 handle.
  • A deterministic test proves that a second destination open/write succeeds immediately after timeout cleanup.
  • Hardware closeout runs bash autoresearch rp2040 --rpc-smoke --timeout 120s --skip-lint with a unique daemon port and records the exact decisive output.
  • Conclusions do not label either RP2040 board defective without the artifact, host, and transport controls above.

Fixed environmental constraint / non-goals

  • Acronis must remain installed, enabled, and unmodified.
  • Do not stop, disable, uninstall, upgrade, reconfigure, detach, or mutate any Acronis service, driver, filter, or installation.
  • Read-only observation of the existing storage stack is allowed; changing it is not.
  • Do not run chkdsk or attempt filesystem repair on the synthetic BOOTSEL FAT volume.
  • Do not treat “Restart Manager found no process” as proof that no kernel component participated.
  • Do not require a BOOTSEL button sequence as the final fbuild recovery design.
  • Do not install Zadig or rebind the current machine's RP2040 interfaces as part of this investigation.

External reports

Similar symptoms exist outside fbuild:

These are analogous reports, not proof of the same root cause.

Related issues

Activity

  1. zackees commented on Jul 23, 2026

    @zackees
    MemberAuthor

    Pre-hardening rollback map and live d5144e01 result (2026-07-23)

    Bottom line

    I rebuilt the exact pre-hardening boundary at d5144e01, installed that build into an isolated venv, verified that uv run --no-project fbuild resolved to the rebuilt executable, and used it to deploy the pinned 882,688-byte UF2.

    The pre-hardening deploy also failed on this host/board. It returned a mass-storage error 1006 while copying to G:\NEW.UF2, then the managed-picotool fallback could see the PICOBOOT interface but could not open it because MI_01 has no compatible driver. Windows logged Disk 153 at LBA 0x134 plus thirteen Disk 51 events during this attempt.

    That is evidence that the initial storage-path failure exists before #1083. It is not evidence that the watchdog cleanup is safe: the post-timeout abandoned-worker/handle hazard introduced by the hardening remains a separate issue.

    Which commit to roll back to

    Commit What it represents Use
    0bf8feb0 (v2.5.2) Last published tag before #1083. It predates b82948e5 and d5144e01. Easiest released rollback, but slightly earlier than the exact boundary.
    b82948e5 Last RP2040 behavior commit before hardening: #1077, “avoid inferring QSPI fault from eject timeout.” Best “last RP2040 code before hardening” point.
    d5144e01 Exact main commit immediately before squash merge debc9963. Its only diff from b82948e5 is six lines in CLAUDE.md; there is no fbuild-deploy diff. Recommended clean A/B baseline. This is what I built and ran below.
    a3d8c58b PR-only phase with fresh-enumeration transfer retries. It changes only rp2040.rs and does not contain the later watchdog/full hardening. It was never on main. Useful surgical bisect point after d5144e01; not a release rollback.
    62d86f2a Formatting commit on the PR branch. No useful behavior boundary.
    c472f56f Full PR head: watchdog, topology diagnostics, tunable budgets, fallback/policy changes. Hardened comparison point, not a rollback.
    debc9963 Squash merge of #1083; first main commit containing the complete hardening. First mainline “after” point.
    d5eb1524 (v2.5.3) Release tag that contains debc9963 plus subsequent changes. Not a pre-hardening build.

    The intermediate SHAs a3d8c58b and c472f56f initially appear missing from a normal local main checkout because #1083 was squash-merged. Fetch the PR head before using either:

    git fetch origin pull/1083/head:refs/remotes/origin/pr/1083

    Rebuild and run the exact pre-hardening CLI

    This keeps the old CLI/daemon out of the global install and makes uv run --no-project fbuild resolve to the rebuilt executable:

    git worktree add --detach .claude/worktrees/issue-1161-d5144e01 `
      d5144e0111d04b390f431e9700211a65cfee2f9d
    Set-Location .claude/worktrees/issue-1161-d5144e01
    
    uv venv .venv
    uv pip install --python .venv\Scripts\python.exe "setuptools>=64"
    
    $env:VIRTUAL_ENV = (Resolve-Path .venv).Path
    $env:PATH = "$env:VIRTUAL_ENV\Scripts;$env:PATH"
    $env:FBUILD_BUILD_RELEASE = "1"
    
    # On this host, plain `uv build --wheel` failed to provision setuptools in
    # the historical isolated build env. This explicit fallback succeeded.
    uv build --wheel --no-build-isolation --out-dir dist-issue1161-d514
    uv pip install --python .venv\Scripts\python.exe --force-reinstall `
      .\dist-issue1161-d514\fbuild-2.5.2-cp313-cp313-win_amd64.whl
    
    uv run --no-project fbuild --version
    # fbuild 2.5.2

    Provenance from this build:

    • source commit: d5144e0111d04b390f431e9700211a65cfee2f9d
    • wheel: 26,199,746 bytes, SHA-256 B96AF74A035DE35925D931CC9A775185F71371D5EF33A48F85E2610EF8D140DC
    • CLI: 35,405,824 bytes, SHA-256 36CDCC27AAE44740956BABFA2B7F56A01E7CC75B6680B9076AB4D447D43A79B5
    • daemon: 37,073,408 bytes, SHA-256 F78A756C736D44D72D1089B28590182210AA7D4E55761D6CC6E6C1D4BCE412D3
    • where.exe fbuild under uv run --no-project listed the isolated .venv\Scripts\fbuild.exe first.

    For isolation, I used:

    $env:FBUILD_DAEMON_PORT = "52161"
    $env:FBUILD_DEV_MODE = "1"
    $env:FBUILD_CACHE_DIR = "<worktree>\.claude\tmp\issue1161-runtime\cache"
    $env:FBUILD_BUILD_DIR = "<worktree>\.claude\tmp\issue1161-runtime\build"

    Artifact-controlled deploy command

    The pinned artifact from this issue was recovered at:

    C:\Users\niteris\dev\fastled\.build\pio\rp2040\.fbuild\build\release\firmware.uf2
    

    Verified values:

    • firmware.uf2: 882,688 bytes, SHA-256 1C770999678BB7563A81D36C696DC44C074A712E25F0DE68A8A83F859E8AF0E0
    • firmware.bin: 441,316 bytes, SHA-256 F29910E24324F71ADB58D40DE97D6C1746F1F688BE111615858CD01F70B91BC7

    I copied the existing firmware.bin, .uf2, .elf, .map, and build_info.json byte-for-byte into the isolated $env:FBUILD_BUILD_DIR\release and rehashed them. The deploy used --skip-build, so the artifact could not change:

    uv run --no-project fbuild `
      C:\Users\niteris\dev\fastled\.build\pio\rp2040 `
      deploy -e rp2040 --skip-build -v

    The command ran behind a 120-second outer process boundary. Before starting it there were no fbuild.exe or fbuild-daemon.exe processes. The boundary did not fire; the historical deploy returned exit 1 itself.

    Decisive output:

    deploy failed: failed to copy RP2040 UF2 ...\firmware.uf2 to G:\NEW.UF2:
    The volume for a file has been externally altered so that the opened file
    is no longer valid. (os error 1006)
    
    Managed picotool error: ... RP2040 device in BOOTSEL mode, but picotool was
    unable to connect. You may need to install a driver via Zadig.
    

    No driver was installed or rebound, and Acronis was not stopped, modified, or inspected beyond the allowed read-only host observations.

    Hardware and Windows evidence

    Before the attempt:

    • BOOTSEL composite: USB\VID_2E8A&PID_0003\E0C9125B0D9B
    • mass storage MI_00: present/OK
    • PICOBOOT MI_01: present, problem code 28/no driver
    • RPI-RP2: G:, FAT, Healthy/OK
    • runtime identity from the prior enumeration: 2E8A:000A, serial 5303284720C4641C, COM12 (not present while in BOOTSEL)

    During the pre-hardening attempt (start 2026-07-23T12:55:09-07:00):

    • Disk 153: one retry at LBA 0x134 on Disk 2
    • Disk 51: thirteen paging-operation errors on \Device\Harddisk2\DR25
    • G:\NEW.UF2 was not present after the failure

    Immediately after the command, BOOTSEL MI_00/MI_01 and G: were still present, G: still reported Healthy/OK, and no fbuild CLI/daemon process remained. Thus this particular old-build failure did not leave a live process-level handle after it returned.

    Later G: disappeared. After the board was physically replugged, Windows did not enumerate either 2E8A endpoint. The only new present node was:

    USB\VID_0000&PID_0002\6&3AF0F9CE&0&14
    Unknown USB Device (Device Descriptor Request Failed)
    problem code 43
    Port_#0014.Hub_#0001
    ACPI(...RHUB)#ACPI(PO14)   # direct root port
    

    All prior 2E8A nodes and COM12 were non-present; the current scanner showed only the phantom COM12 record. This is a post-replug descriptor-level enumeration failure. It is recorded here without asserting that the deploy caused it.

    Interpretation and next bisect

    1. d5144e01 failed on the same Windows mass-storage surface with the same pinned artifact. Therefore harden(rp2040): hub-path deploy robustness — retries, picotool fallback, topology diagnostics, tunable budgets #1083 is not required to produce the initial failure.
    2. This run did not exercise run_with_watchdog, because it predates that code. It cannot clear the abandoned-worker/handle concern in current builds.
    3. The sharp next code boundary is a3d8c58b: it has fresh-enumeration retries but no watchdog. After host/USB enumeration is restored, run that against the same pinned files and an isolated port/cache.
    4. Then compare c472f56f or debc9963 with the same process boundary. If only the watchdog-enabled case leaves a live daemon worker/handle after its caller returns, report that separately from the underlying Disk 51/153 transport failure.
    5. bash autoresearch rp2040 --rpc-smoke --timeout 120s --skip-lint could not be completed because this deploy did not produce an application CDC endpoint.

    One additional limitation: this was started with a clean fbuild process audit, but not after a full host reboot. It is a useful pre-hardening reproduction, not the complete clean-reboot A/B matrix requested by the issue.

  2. zackees commented on Jul 23, 2026

    @zackees
    MemberAuthor

    Open-source uploader comparison: make PICOBOOT a first-class RP2040 transport

    The Arduino/PlatformIO source comparison changes the recommended direction here.

    What the other toolchains actually do

    Arduino IDE itself does not own one upload implementation; the selected board core supplies the recipe.

    • The Earle Philhower Arduino-Pico core used by fbuild defaults to its UF2-volume path: touch the application serial port, find a BOOTSEL volume, then write NEW.UF2. Source: uf2conv.py.
    • That same core also exposes a Picotool upload method which runs picotool load <firmware.uf2> -f -x, bypassing the Windows filesystem. Source: platform.txt.
    • Arduino's official rp2040tools uses rp2040load to orchestrate elf2uf2, picotool info, picotool load, and picotool reboot. Its sequence is visible directly in main.go.
    • PlatformIO's default RP2040 protocol is picotool. Its builder performs the 1200-baud touch and then launches rp2040load -v -D <firmware.elf>; this is direct PICOBOOT transport, not an RPI-RP2 disk copy. Source: platform-raspberrypi/builder/main.py.

    So the useful comparison is:

    Path USB interface Transfer
    Arduino-Pico default / current fbuild primary BOOTSEL MI_00 mass storage Write NEW.UF2 through Windows filesystem/storage stack
    PlatformIO default / Arduino Picotool option BOOTSEL MI_01 PICOBOOT picotool load directly over USB

    Recommended fbuild direction

    fbuild should make the direct PICOBOOT/picotool path a first-class, explicitly selectable transport and preferably the primary automatic path, rather than reaching it only after the mass-storage copy has already failed.

    The model already proven by PlatformIO/Arduino is approximately:

    1. If application CDC is present, perform the 1200-baud reset/touch.
    2. Wait for the 2E8A:0003 BOOTSEL/PICOBOOT interface.
    3. Probe with picotool info.
    4. Load the generated UF2/ELF using picotool load.
    5. Reboot/execute and verify application re-enumeration.
    6. Retain RPI-RP2/NEW.UF2 as an explicit alternative or fallback transport.

    fbuild already has a managed-picotool fallback, so this should mostly be a matter of transport selection, ordering, error classification, and testing—not implementing PICOBOOT from scratch.

    This also avoids the exact Windows surface implicated in this reproduction: the pre-hardening failure was in the G:\NEW.UF2 filesystem write and produced Disk 51/153 events. A successful direct PICOBOOT load would never open that drive or send the firmware through the Windows mass-storage filesystem path.

    Important limitation from this host

    The historical fbuild run did reach managed picotool, but Windows currently reports PICOBOOT MI_01 with problem code 28/no compatible driver, so picotool could see BOOTSEL but could not open it. That is a separate connection/driver prerequisite and does not make the mass-storage route equivalent to PICOBOOT. No driver was installed or rebound during this investigation.

    Once USB enumeration is restored, the next controlled test should be PICOBOOT-only against the already-pinned firmware artifact. If that succeeds without touching G: and without new Disk 51/153 events, it cleanly isolates the observed failure to the mass-storage deployment path and provides the implementation direction for fbuild.

  3. zackees commented on Jul 25, 2026

    @zackees
    MemberAuthor

    Hardware result and revised zero-install transport direction (2026-07-25)

    The direct-FAT experiment now has a decisive SRAM-vs-flash control on the attached RP2040:

    • The production raw-volume/SCSI writer accepted a one-block SRAM-targeted UF2 through the exact RPI-RP2 FAT partition.
    • A flash-targeted UF2 failed on the first SCSI WRITE(10) sector with Windows error 121 after the BOOTSEL volume had already been locked and dismounted.
    • With WinUSB bound, managed picotool reached the same BOOTSEL device, but an external-flash load failed at 0%. A PICOBOOT SRAM write had succeeded in the control run.
    • Therefore the direct writer itself works and the observed flash failure is below the mounted FAT/file layer. On this attached board, the A/B evidence points to the external QSPI flash path being unavailable; it is not NEW.UF2 file contention.

    This also changes the product recommendation from the earlier “PICOBOOT primary” comment. Raspberry Pi's current Windows instructions explicitly require binding WinUSB to RP2 Boot (Interface 1) (normally via Zadig). Making picotool primary would impose a driver-install step on users who can already use the stock zero-install UF2 volume.

    The implementation direction under test is now:

    1. Keep stock UF2 mass-storage as the default (no install, no UAC).
    2. If that write fails on Windows, return a typed exact-volume action to the CLI.
    3. With explicit --admin, launch one nonce-bound one-shot helper for only the raw-FAT write. Revalidate the artifact SHA-256/length/family/target, exact RPI-RP2 root, physical-disk mapping, RP2040 MBR, and FAT geometry after elevation. Never elevate the daemon/build/cache.
    4. Keep picotool explicitly selectable for machines where WinUSB is already configured; do not make it a prerequisite.

    Software evidence on the local branch so far: 92 RP2040 tests pass (3 attached-hardware tests ignored), the scoped-helper/core tests pass, and Clippy passes across the affected CLI/core/daemon/deploy crates. Live end-to-end execution remains pending a physical BOOTSEL reset because the SRAM control intentionally left the board with no USB runtime and the current host sees only the old phantom COM12 record plus a descriptor-error node on the direct root port.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions