Repository navigation
Description
Context
Two RP2040 boards have now refused programming. The current failure has a specific, repeatable state:
- The RP2040 enters BOOTSEL and enumerates as
USB\VID_2E8A&PID_0003\E0C9125B0D9B. - Its mass-storage interface (
MI_00) is healthy. - Windows mounts
RPI-RP2onG:and reports the volume healthy. INFO_UF2.TXTand other reads succeed.- Writes do not complete:
- fbuild's
NEW.UF2write stalls; - a normal Windows file copy stalls;
- a paced write-through test using 8 KiB chunks stalled before its first completed chunk and recorded zero progress.
- fbuild's
- Windows logged Disk events 51 and 153 during the attempts, including retries at LBAs
0x134and0x14c. - Restart Manager reported no user-mode holder, and an exclusive volume open succeeded between attempts.
The last point does not prove that the failure is device-side. Kernel storage filters, filesystem behavior, USB transport faults, QSPI programming behavior, and a host/controller problem can all exist without a user-mode volume handle. A USB trace is required before claiming that the RP2040 bulk-OUT endpoint failed to acknowledge the transfer.
The separate MI_01 PICOBOOT interface currently has no Windows driver. That does not explain why the healthy MI_00 mass-storage interface can be read but not written.
fbuild regression question
RP2040 deploy hardening was merged in commit debc9963 through PR #1083.
The relevant commit boundary is:
d5144e0111d04b390f431e9700211a65cfee2f9d— immediately before the hardeninga3d8c58b94111a4e055f66d67d4ff8ad702fd730— fresh-enumeration retriesc472f56f5989a4507321e862fdb62d7c1bcf309f— watchdog and broader hardeningdebc9963527e0a5a09f1224883b02d63ba174425— merged hardening16bb9ffae6aeab38957176357425110b002d98b1— currentmainduring this reproduction
Static comparison shows that the low-level first-write shape was not materially changed by #1083: both sides open/truncate NEW.UF2 and make a whole-buffer write_all call.
The hardening did introduce a definite recovery hazard. run_with_watchdog deliberately abandons a timed-out worker thread:
fbuild/crates/fbuild-deploy/src/rp2040.rs
Lines 544 to 560 in 16bb9ff
| /// Run `work` on a dedicated thread and give up after `budget`. A storport | |
| /// retry storm behind a sick hub can block the NEW.UF2 `write_all` for | |
| /// minutes with no output; this turns that hang into an actionable failure | |
| /// that feeds the fresh-enumeration retry gate. On timeout the worker thread | |
| /// is deliberately abandoned, not joined: it only touches its own locals and | |
| /// its channel send fails silently once this receiver is dropped, so the | |
| /// wedged kernel write can unblock (or not) without holding up the deploy. | |
| fn run_with_watchdog<T: Send + 'static>( | |
| budget: Duration, | |
| label: &str, | |
| work: impl FnOnce() -> Result<T> + Send + 'static, | |
| ) -> Result<T> { | |
| let (sender, receiver) = std::sync::mpsc::channel(); | |
| std::thread::spawn(move || { | |
| let _ = sender.send(work()); | |
| }); | |
| match receiver.recv_timeout(budget) { |
That worker owns the open NEW.UF2 file. If the kernel write remains blocked, the thread and handle can remain alive inside the daemon after fbuild reports a timeout. This can contaminate retries and later deploy attempts even if the watchdog did not cause the original first-write stall.
The existing watchdog test verifies that the caller returns quickly by leaving a five-second worker running. It does not verify worker termination, handle release, or whether an immediate second write can open the destination.
There is also a local version-isolation risk:
- Global executable:
C:\tools\python13\Scripts\fbuild.exe, reporting2.5.2 - FastLED virtual-environment executable: reporting
2.5.3 - Installed Python metadata in that environment: reporting
2.5.2
Every hardware result therefore needs to record the exact executable, executable hash, source commit, daemon port, and daemon binary used.
Pinned artifact evidence
The current artifact is:
- Size:
882,688bytes - SHA-256:
1C770999678BB7563A81D36C696DC44C074A712E25F0DE68A8A83F859E8AF0E0 - 1,724 UF2 blocks of 512 bytes
- RP2040 family ID:
0xE48BFF56 - Target range:
0x10000000through0x1006BB00 - Block magic, block numbering, payload lengths, family ID, and declared block count are structurally valid
An earlier deploy transcript reported a different artifact size. Commit comparisons must therefore use one pinned artifact without rebuilding between runs.
Proposal
1. Produce a deterministic reproduction record
Before every attempt, record:
- physical board serial and VID:PID;
- PnP health for both BOOTSEL interfaces;
- drive letter and USB hub/root-port topology;
- exact fbuild executable path, version, hash, source commit, and daemon binary;
- a unique five-digit
FBUILD_DAEMON_PORT; - artifact path, size, and SHA-256;
- relevant Windows Disk 51/153 events before and after the attempt;
- whether any fbuild worker/process/volume handle remains after timeout.
Use an outer killable process boundary for any experiment that may block. Do not allow one timed-out attempt to contaminate the next case.
2. Run a transport/artifact matrix
On the same board, cable, direct motherboard port, and clean BOOTSEL enumeration, compare:
- the previously proven RAM-only UF2;
- a small official known-good RP2040 flash Blink UF2;
- the pinned fbuild-generated flash UF2.
Interpretation:
- RAM succeeds but both flash images fail: investigate the flash-target/QSPI path.
- Official flash succeeds but fbuild flash fails: investigate artifact generation or layout.
- All three fail on the first storage request: investigate the Windows/USB/volume path.
- The result changes with fbuild commit while the pinned artifact and physical setup remain identical: investigate an fbuild deploy regression.
Also repeat the decisive case using:
- a known-good data cable;
- a direct USB 2.0-capable motherboard port;
- a second clean host, preferably Linux.
On Linux, a guarded comparison between a normal mounted-volume copy and an unmounted raw block-device write may distinguish filesystem integration from USB/device behavior. The exact block device must be verified before any raw write.
3. Capture the failing layer
Capture one failure with:
- USBPcap/Wireshark, to determine whether the first mass-storage bulk-OUT transaction leaves the host and what response/reset sequence follows;
- Process Monitor filtered to the selected volume and fbuild process;
- Windows Disk, StorPort, Kernel-PnP, and USB events;
- read-only
fltmc/PnP inspection when elevated access is available.
The trace should classify the failure without assuming that readable media plus a missing user-mode handle proves a board fault.
4. Perform an exact hardening A/B
Build and run isolated versions at the commit boundary listed above:
- use separate worktrees/installations;
- use separate daemon ports and state directories;
- verify both CLI and daemon provenance;
- use the same pinned UF2;
- start with the pre-hardening build after a clean reboot;
- do not run an older build after a watchdog failure unless every newer worker/process/handle has been proven gone.
Decision criteria:
d5144e01repeatedly succeeds whilec472f56ffails in the clean identical setup: hardening regression.- Both fail on the same first storage operation: the initial stall predates or is independent of the hardening.
- Both fail initially, but only watchdog-enabled builds retain a worker/handle or poison the next attempt: underlying transport failure plus a separate fbuild recovery regression.
5. Make timeout cleanup handle-safe
If the watchdog path is retained, it must not abandon a thread that owns a removable-volume file handle.
Use either:
- a killable helper process whose termination closes the handle; or
- cancellable Windows overlapped I/O using
CancelIoEx, followed by bounded completion and thread joining.
After timeout:
- the worker must be terminated or joined;
- the
NEW.UF2handle must be closed; - stale volume paths must be discarded;
- retries may occur only after a verified fresh BOOTSEL enumeration;
- an immediate second open/write probe must not fail because of the prior attempt.
Add a deterministic regression test covering timeout followed by immediate handle reacquisition and a second write.
6. Use PICOBOOT only as a controlled comparison
PICOBOOT/picotool can be used as a transport control on another host, or where the interface already has a compatible driver. Do not install or rebind drivers on the current machine as part of this issue.
Acceptance criteria
- A reproducible log captures exact board, artifact, executable, daemon, USB topology, and Windows event identities.
- The RAM-only, official flash, and fbuild flash artifact matrix is completed without rebuilding between cases.
- At least one USB trace identifies whether the first write leaves the host and what fails next.
- The pre-hardening through current-main A/B identifies whether harden(rp2040): hub-path deploy robustness — retries, picotool fallback, topology diagnostics, tunable budgets #1083 changed the initial failure.
- The initial transport failure and any post-timeout contamination are reported as separate findings.
- A watchdog timeout leaves no live worker, helper, or
NEW.UF2handle. - A deterministic test proves that a second destination open/write succeeds immediately after timeout cleanup.
- Hardware closeout runs
bash autoresearch rp2040 --rpc-smoke --timeout 120s --skip-lintwith a unique daemon port and records the exact decisive output. - Conclusions do not label either RP2040 board defective without the artifact, host, and transport controls above.
Fixed environmental constraint / non-goals
- Acronis must remain installed, enabled, and unmodified.
- Do not stop, disable, uninstall, upgrade, reconfigure, detach, or mutate any Acronis service, driver, filter, or installation.
- Read-only observation of the existing storage stack is allowed; changing it is not.
- Do not run
chkdskor attempt filesystem repair on the synthetic BOOTSEL FAT volume. - Do not treat “Restart Manager found no process” as proof that no kernel component participated.
- Do not require a BOOTSEL button sequence as the final fbuild recovery design.
- Do not install Zadig or rebind the current machine's RP2040 interfaces as part of this investigation.
External reports
Similar symptoms exist outside fbuild:
- Raspberry Pi forum: readable BOOTSEL files but writes disconnect before transferring bytes. A Linux unmount/raw-write comparison worked in that report, suggesting the mounted filesystem path can matter.
- Adafruit: Windows UF2 copy hangs at 0%. Their documented case involved host disk-management software, showing that this surface symptom is not sufficient to diagnose a board fault.
- Raspberry Pi: the BOOTSEL drive is a synthetic virtual volume.
- UF2 specification.
These are analogous reports, not proof of the same root cause.
Related issues
- investigate(rp2040): harden hub-path BOOTSEL mass-storage diagnostics #1081 — hub-path BOOTSEL mass-storage diagnostics
- harden(rp2040): deploy robustness across host setups (hub chains, drive-scan stalls, timeout policy) #1082 — deploy hardening and the original open question about abandoning blocked writer threads
- feat(windows): recover RP2040 phantom/BOOTSEL USB state through scoped admin PnP flow #1152 — Windows phantom/BOOTSEL recovery; distinct because the present volume is healthy and readable
- feat(rp2040): zero-touch deployment to an as-received Pico via ROM UF2 #1040 — original zero-touch RP2040 deployment and RAM-versus-flash evidence
- feat(rp2040): deploy parity with Arduino-Pico, PlatformIO, and official picotool #1049 — canonical RP2040 UF2 artifact and transfer semantics
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsTriage
Activity
zackees commentedon Jul 23, 2026
Pre-hardening rollback map and live
d5144e01result (2026-07-23)Bottom line
I rebuilt the exact pre-hardening boundary at
d5144e01, installed that build into an isolated venv, verified thatuv run --no-project fbuildresolved to the rebuilt executable, and used it to deploy the pinned 882,688-byte UF2.The pre-hardening deploy also failed on this host/board. It returned a mass-storage error 1006 while copying to
G:\NEW.UF2, then the managed-picotool fallback could see the PICOBOOT interface but could not open it becauseMI_01has no compatible driver. Windows logged Disk 153 at LBA0x134plus thirteen Disk 51 events during this attempt.That is evidence that the initial storage-path failure exists before #1083. It is not evidence that the watchdog cleanup is safe: the post-timeout abandoned-worker/handle hazard introduced by the hardening remains a separate issue.
Which commit to roll back to
0bf8feb0(v2.5.2)b82948e5andd5144e01.b82948e5d5144e01maincommit immediately before squash mergedebc9963. Its only diff fromb82948e5is six lines inCLAUDE.md; there is nofbuild-deploydiff.a3d8c58brp2040.rsand does not contain the later watchdog/full hardening. It was never onmain.d5144e01; not a release rollback.62d86f2ac472f56fdebc9963maincommit containing the complete hardening.d5eb1524(v2.5.3)debc9963plus subsequent changes.The intermediate SHAs
a3d8c58bandc472f56finitially appear missing from a normal localmaincheckout because #1083 was squash-merged. Fetch the PR head before using either:Rebuild and run the exact pre-hardening CLI
This keeps the old CLI/daemon out of the global install and makes
uv run --no-project fbuildresolve to the rebuilt executable:Provenance from this build:
d5144e0111d04b390f431e9700211a65cfee2f9dB96AF74A035DE35925D931CC9A775185F71371D5EF33A48F85E2610EF8D140DC36CDCC27AAE44740956BABFA2B7F56A01E7CC75B6680B9076AB4D447D43A79B5F78A756C736D44D72D1089B28590182210AA7D4E55761D6CC6E6C1D4BCE412D3where.exe fbuildunderuv run --no-projectlisted the isolated.venv\Scripts\fbuild.exefirst.For isolation, I used:
Artifact-controlled deploy command
The pinned artifact from this issue was recovered at:
Verified values:
firmware.uf2: 882,688 bytes, SHA-2561C770999678BB7563A81D36C696DC44C074A712E25F0DE68A8A83F859E8AF0E0firmware.bin: 441,316 bytes, SHA-256F29910E24324F71ADB58D40DE97D6C1746F1F688BE111615858CD01F70B91BC7I copied the existing
firmware.bin,.uf2,.elf,.map, andbuild_info.jsonbyte-for-byte into the isolated$env:FBUILD_BUILD_DIR\releaseand rehashed them. The deploy used--skip-build, so the artifact could not change:The command ran behind a 120-second outer process boundary. Before starting it there were no
fbuild.exeorfbuild-daemon.exeprocesses. The boundary did not fire; the historical deploy returned exit 1 itself.Decisive output:
No driver was installed or rebound, and Acronis was not stopped, modified, or inspected beyond the allowed read-only host observations.
Hardware and Windows evidence
Before the attempt:
USB\VID_2E8A&PID_0003\E0C9125B0D9BMI_00: present/OKMI_01: present, problem code 28/no driverRPI-RP2:G:, FAT, Healthy/OK2E8A:000A, serial5303284720C4641C,COM12(not present while in BOOTSEL)During the pre-hardening attempt (start
2026-07-23T12:55:09-07:00):0x134on Disk 2\Device\Harddisk2\DR25G:\NEW.UF2was not present after the failureImmediately after the command, BOOTSEL
MI_00/MI_01andG:were still present,G:still reported Healthy/OK, and no fbuild CLI/daemon process remained. Thus this particular old-build failure did not leave a live process-level handle after it returned.Later
G:disappeared. After the board was physically replugged, Windows did not enumerate either2E8Aendpoint. The only new present node was:All prior
2E8Anodes andCOM12were non-present; the current scanner showed only the phantomCOM12record. This is a post-replug descriptor-level enumeration failure. It is recorded here without asserting that the deploy caused it.Interpretation and next bisect
d5144e01failed on the same Windows mass-storage surface with the same pinned artifact. Therefore harden(rp2040): hub-path deploy robustness — retries, picotool fallback, topology diagnostics, tunable budgets #1083 is not required to produce the initial failure.run_with_watchdog, because it predates that code. It cannot clear the abandoned-worker/handle concern in current builds.a3d8c58b: it has fresh-enumeration retries but no watchdog. After host/USB enumeration is restored, run that against the same pinned files and an isolated port/cache.c472f56fordebc9963with the same process boundary. If only the watchdog-enabled case leaves a live daemon worker/handle after its caller returns, report that separately from the underlying Disk 51/153 transport failure.bash autoresearch rp2040 --rpc-smoke --timeout 120s --skip-lintcould not be completed because this deploy did not produce an application CDC endpoint.One additional limitation: this was started with a clean fbuild process audit, but not after a full host reboot. It is a useful pre-hardening reproduction, not the complete clean-reboot A/B matrix requested by the issue.
zackees commentedon Jul 23, 2026
Open-source uploader comparison: make PICOBOOT a first-class RP2040 transport
The Arduino/PlatformIO source comparison changes the recommended direction here.
What the other toolchains actually do
Arduino IDE itself does not own one upload implementation; the selected board core supplies the recipe.
NEW.UF2. Source:uf2conv.py.picotool load <firmware.uf2> -f -x, bypassing the Windows filesystem. Source:platform.txt.rp2040toolsusesrp2040loadto orchestrateelf2uf2,picotool info,picotool load, andpicotool reboot. Its sequence is visible directly inmain.go.picotool. Its builder performs the 1200-baud touch and then launchesrp2040load -v -D <firmware.elf>; this is direct PICOBOOT transport, not anRPI-RP2disk copy. Source:platform-raspberrypi/builder/main.py.So the useful comparison is:
MI_00mass storageNEW.UF2through Windows filesystem/storage stackMI_01PICOBOOTpicotool loaddirectly over USBRecommended fbuild direction
fbuild should make the direct PICOBOOT/picotool path a first-class, explicitly selectable transport and preferably the primary automatic path, rather than reaching it only after the mass-storage copy has already failed.
The model already proven by PlatformIO/Arduino is approximately:
2E8A:0003BOOTSEL/PICOBOOT interface.picotool info.picotool load.RPI-RP2/NEW.UF2as an explicit alternative or fallback transport.fbuild already has a managed-picotool fallback, so this should mostly be a matter of transport selection, ordering, error classification, and testing—not implementing PICOBOOT from scratch.
This also avoids the exact Windows surface implicated in this reproduction: the pre-hardening failure was in the
G:\NEW.UF2filesystem write and produced Disk 51/153 events. A successful direct PICOBOOT load would never open that drive or send the firmware through the Windows mass-storage filesystem path.Important limitation from this host
The historical fbuild run did reach managed picotool, but Windows currently reports PICOBOOT
MI_01with problem code 28/no compatible driver, sopicotoolcould see BOOTSEL but could not open it. That is a separate connection/driver prerequisite and does not make the mass-storage route equivalent to PICOBOOT. No driver was installed or rebound during this investigation.Once USB enumeration is restored, the next controlled test should be PICOBOOT-only against the already-pinned firmware artifact. If that succeeds without touching
G:and without new Disk 51/153 events, it cleanly isolates the observed failure to the mass-storage deployment path and provides the implementation direction for fbuild.zackees commentedon Jul 25, 2026
Hardware result and revised zero-install transport direction (2026-07-25)
The direct-FAT experiment now has a decisive SRAM-vs-flash control on the attached RP2040:
RPI-RP2FAT partition.SCSI WRITE(10)sector with Windows error 121 after the BOOTSEL volume had already been locked and dismounted.NEW.UF2file contention.This also changes the product recommendation from the earlier “PICOBOOT primary” comment. Raspberry Pi's current Windows instructions explicitly require binding WinUSB to RP2 Boot (Interface 1) (normally via Zadig). Making picotool primary would impose a driver-install step on users who can already use the stock zero-install UF2 volume.
The implementation direction under test is now:
--admin, launch one nonce-bound one-shot helper for only the raw-FAT write. Revalidate the artifact SHA-256/length/family/target, exactRPI-RP2root, physical-disk mapping, RP2040 MBR, and FAT geometry after elevation. Never elevate the daemon/build/cache.Software evidence on the local branch so far: 92 RP2040 tests pass (3 attached-hardware tests ignored), the scoped-helper/core tests pass, and Clippy passes across the affected CLI/core/daemon/deploy crates. Live end-to-end execution remains pending a physical BOOTSEL reset because the SRAM control intentionally left the board with no USB runtime and the current host sees only the old phantom COM12 record plus a descriptor-error node on the direct root port.