harden(rp2040): deploy robustness across host setups (hub chains, drive-scan stalls, timeout policy) #1082

Related: #1081 (hub-path BOOTSEL diagnostics), #1040 (stock-ROM first deploy), #1049 (deploy parity). Parent burn-down: FastLED/FastLED#3648.

Context

#1081's investigation identified the hub-vs-direct differential mechanisms (RP2040-E5 busy-bus reset erratum unpatchable in mask ROM, VBUS sag during QSPI program, TT bulk stalls escalating to port resets). A first hardening slice now exists locally (bounded UF2 transfer retries gated on a fresh ROM enumeration, plus managed-picotool fallback when the BOOTSEL volume never mounts) and was validated with two consecutive hub-path deploys on the reference board (2E8A:000A, serial 5303284720C4641C, COM12): full flash, CDC return, and firmware serial output observed (Hello from RP2040 Pico!). Notably, both runs logged a transient CDC error during the 1200-bps touch that the existing BOOTSEL-appeared signal rode through.

This issue inventories what still goes out of spec on other machines — deeper hub chains, different timings, enterprise policies, first-plug driver installs — and proposes a prioritized hardening plan. Datasheet references: FastLED/datasheets → raspberrypi/mcu/RP2040/SOURCES.md (RP2040 datasheet incl. erratum RP2040-E5; hardware-design-with-rp2040).

The deploy chain and its current budget

Every RP2040/RP2350 deploy walks this chain (crates/fbuild-deploy/src/rp2040.rs):

Failure-mode inventory on foreign setups

Stage 4 — volume discovery (most fragile on foreign machines)

• Mapped network drives poison the A–Z scan. volume_roots() probes every letter with fs::read_to_string + read_dir; a disconnected SMB mapping can block each probe tens of seconds, eating the whole 10 s window before the real BOOTSEL letter is reached. Fix: GetDriveType() == DRIVE_REMOVABLE filter before touching a root.
• No drive letter at all (GPO letter-hiding, exhaustion, mount-as-folder): volume mounts but never appears at a root. Fix: FindFirstVolume-style enumeration, or treat "PnP shows USB\VID_2E8A&PID_0003 present but no volume" as a distinct state → go straight to picotool.
• First-plug driver install exceeds the window. Windows re-runs device install per hub-port instance path; "searching Windows Update" can add 10–60 s. Applies to stage 7's 15 s too. Fix: env-tunable timeouts and/or gate on PnP device arrival instead of a fixed window.
• macOS 13+ accessory approval (Apple Silicon): first-seen USB device needs a user click before enumeration — guaranteed timeout with zero diagnostic. Fix: platform-specific hint in the timeout message.

Stages 3–5 — hub topology and timing

• Hubs-on-hubs multiply enumeration latency. Up to 5 external tiers; ≥100 ms attach debounce per tier plus host retry cycles, and the deploy does two full re-enumerations (runtime → BOOTSEL → runtime). A flaky 2-tier chain can blow either window even when the device eventually arrives.
• FS hub segments / chained single-TT hubs enlarge the RP2040-E5 blast radius. B0/B1 silicon needs 800 µs of idle bus after reset; on a broadcast FS segment a polled sibling (keyboard, serial dongle at 30–50 µs intervals) can make BOOTSEL enumeration never succeed; the boot ROM cannot be patched. Retries only help if re-enumeration eventually wins the race. Fix: topology capture (CfgMgr32 parent walk) so failures can say "board is 2 hubs deep with 3 siblings — move to a direct port" (investigate(rp2040): harden hub-path BOOTSEL mass-storage diagnostics #1081's diagnostic slice).
• Cascaded bus-powered hubs are out of power spec. QSPI erase/program current spikes brown out marginal VBUS mid-copy, possibly repeatably (retry won't converge). Same topology capture can flag "bus-powered hub chain" preemptively.
• KVM switches and monitor/dock hubs (DisplayLink and friends) re-enumerate children unpredictably; a KVM focus switch mid-transfer is a guaranteed kill. Only diagnosable via topology strings.

Stage 5 — the write itself

• No watchdog on the copy. A storport retry storm behind a sick hub can block write_all for minutes with no output; the daemon just hangs. Fix: per-attempt watchdog (~60 s) that abandons the blocked thread and routes to the retry/fallback path.
• GPO/BitLocker removable-write-deny returns error 5 (ACCESS_DENIED) — currently the generic message. Fix: typed guidance like the existing 121/1006/1392 branches.

Stage 7 — the false-failure trap

• After a confirmed eject (ROM accepted the image), a CDC timeout still fails the whole deploy even though the flash succeeded and the sketch runs. On slow-driver-install machines this makes CI re-flash a healthy board in a loop. Fix: distinct outcome ("flashed, CDC unconfirmed") instead of a hard error.

Stages 1 & 8 — environment dependencies

• Fresh/offline machine: ensure_verified_usb_profiles refuses to deploy if the FastLED/boards USB profile cache was never populated — a network dependency before the board is touched.
• picotool fallback needs WinUSB bound to the RP2 Boot interface on RP2040 (RP2350's ROM ships MS OS 2.0 descriptors; RP2040's does not) — on machines that never ran Zadig/pico-setup the fallback fails cleanly but uselessly. Windows-ARM64 availability of the managed picotool package is unverified.
• Multi-board rigs: two boards in BOOTSEL → select_single_uf2_volume hard-errors. Fix: snapshot volumes before the touch and attribute the one that appeared.

Prioritized hardening plan

P0

• Drive-type filter (DRIVE_REMOVABLE) in the Windows volume scan; never probe remote/fixed drives.
• Topology capture via CfgMgr32 parent walk (root port vs hub tiers, bus- vs self-powered, sibling count) folded into stage-4/5 failure messages — the investigate(rp2040): harden hub-path BOOTSEL mass-storage diagnostics #1081 diagnostic slice.

P1

• Per-attempt write watchdog (~60 s) feeding the existing retry/fallback path.
• Env-tunable stage-4/stage-7 timeouts; consider PnP-arrival gating for stage 4.
• "Flashed, CDC unconfirmed" outcome after confirmed eject instead of deploy failure.

P2

• Appeared-volume attribution for multi-board rigs (pre/post-touch volume diff).
• Typed error-5 (removable write-deny policy) guidance.
• macOS accessory-approval hint on stage-4 timeout.
• Verify managed picotool availability on Windows ARM64; document the RP2040 WinUSB binding caveat in the fallback error.

Acceptance criteria

• A machine with a dead mapped network drive completes stage 4 inside the normal window.
• A stage-4/5 failure behind a hub reports hub depth, power mode, and sibling count when Windows can supply them, and labels unavailable topology explicitly (no guessing).
• A blocked UF2 write terminates within the watchdog budget and produces an actionable typed error.
• A deploy whose eject was observed never reports hard failure solely because the runtime CDC missed the window.
• Focused deterministic tests cover the new branches (drive filter decision, watchdog expiry, CDC-unconfirmed outcome, multi-volume attribution), consistent with the existing injected-closure test style in rp2040.rs.
• No PlatformIO, raw serial, or direct flashing-tool invocation is introduced (fbuild remains the only deploy interface).

Open questions

• Should stage-4 discovery switch entirely to volume enumeration + PnP arrival (removing the drive-letter scan), or keep the scan with the removable filter as the conservative first step?
• Is abandoning a blocked writer thread acceptable in the daemon's threading model, or does the watchdog need a helper-process write?
• Where should the topology facts live so both the deploy error path and fbuild device introspection can render them (fbuild-serial vs fbuild-deploy)?