Repository navigation
Description
Intro
Audit of crates/fbuild-deploy/ for blocking operations without a wall-clock cap. This crate spawns external flash tools (esptool, avrdude, lpc21isp, teensy_loader_cli) and, when the espflash-native feature is on, calls into the espflash library directly to talk to ESP32 family chips over USB-CDC. Every flash invocation here is the kind of operation that can hang on hardware misbehavior (board disconnected mid-flash, USB stall, stub flasher wedged), so a missing wall-clock cap is a daemon-hang surface.
Good news: every subprocess spawn in this crate (esptool, avrdude, lpc21isp, teensy_loader_cli) goes through fbuild_core::subprocess::run_command(...) with an explicit Some(Duration::from_secs(...)). The subprocess path is well-covered.
Bad news: the native espflash library path (esp32_native/) has no wall-clock cap on the high-level library calls. The per-read serialport timeout (3 s verify / 10 s write) bounds individual UART reads, but espflash's connect() / checksum_md5() / write_bin_to_flash() / reset_after() can do their own internal retry loops on top of those reads — none of which we bound with tokio::time::timeout(...) or a thread-level deadline. On a wedged USB-CDC stack (the exact case that motivated the audit) those calls can spin for minutes before surfacing an error.
The Teensy state machine has its own bounded poll loops (wait_for_new_cdc_port, wait_for_disappearance, probe) — all OK.
Findings
| Severity | File:line | Operation | Why unbounded | Suggested fix |
|---|---|---|---|---|
| CRITICAL | crates/fbuild-deploy/src/esp32_native/write.rs:125 |
flasher.write_bin_to_flash(r.offset, &bytes, &mut bridge) (espflash) |
Per-read serial timeout is 10s, but espflash can retry blocks internally with no overall wall-clock cap. A wedged stub flasher mid-write hangs the /api/deploy handler indefinitely. |
Wrap the whole try_write_deployment_native in a tokio::task::spawn_blocking + tokio::time::timeout(Duration::from_secs(180), …); on timeout, drop the Flasher (closes the serial port) and surface FbuildError::DeployFailed("native write timed out after 180s"). |
| HIGH | crates/fbuild-deploy/src/esp32_native/write.rs:88 |
Flasher::connect(connection, …) (write path, espflash) |
Sync handshake + chip detect + stub upload + baud renegotiate. Stub upload over a flaky USB link can stall for minutes. Per-read 10s timeout doesn't cap the number of stub-upload retries espflash performs. | Same enclosing tokio::time::timeout as the write call covers it; alternatively a tighter inner cap (~30s) just around Flasher::connect. |
| HIGH | crates/fbuild-deploy/src/esp32_native/verify.rs:84 |
Flasher::connect(connection, …) (verify path, espflash) |
Same as write-path connect — handshake / stub upload can hang. Verify is supposed to be "fast skip path" (~6s end-to-end); an unbounded connect makes the fast path slower than a re-flash on bad hardware. | Wrap try_verify_deployment_native in tokio::time::timeout(Duration::from_secs(30), …) and on timeout, fall back to the esptool subprocess path (which already has a 30s cap). |
| HIGH | crates/fbuild-deploy/src/esp32_native/verify.rs:111 |
flasher.checksum_md5(r.offset, bytes.len() as u32) (espflash) |
On-chip FLASH_MD5SUM for up to 3 regions in a loop. Single-region MD5 of a 2.4 MB image is bounded (~3s on chip), but a wedged stub or bad MD5 response could see espflash retry without a cap. |
Covered by the function-level timeout suggested for try_verify_deployment_native. No per-region cap needed. |
| MEDIUM | crates/fbuild-deploy/src/esp32_native/write.rs:155 |
connection.reset_after(use_stub, chip) |
Post-write reset is best-effort and logged on failure, but the call itself is sync and unbounded. A USB-JTAG reset with a hung stub can block. | Wrap in tokio::time::timeout(Duration::from_secs(5), …); on timeout, log warn and return Ok (matches existing "reset_after failed, image is on-device" tolerance). |
| MEDIUM | crates/fbuild-deploy/src/esp32_native/verify.rs:146 |
connection.reset_after(use_stub, chip) |
Same as write path's reset_after. Only runs on the success branch (all_match), so the user-visible impact is "device didn't reboot back into the app after a verify-skip", which is already logged as a warn. |
Same 5s cap as the write path. |
| MEDIUM | crates/fbuild-deploy/src/lib.rs:121-135 |
Deployer::post_deploy_recovery default impl |
Explicit 3s deadline already in place (Instant::now() + Duration::from_secs(3) + 100ms poll). Listed here only for completeness — this is the documented "3s fast-poll" in CLAUDE.md. |
No change needed. |
What was searched
Searched on crates/fbuild-deploy/src/:
Command::new/\.output\(\)/\.wait\(\)/\.spawn\(→ no direct subprocess spawns; everything routes throughfbuild_core::subprocess::run_commandwhich takes an explicitOption<Duration>. Every call site in this crate (avr.rs:123,lpc.rs:217,esp32/deployer.rs:330,671,746,teensy/flash.rs:122) passesSome(...). OK.espflash::/Flasher::/connection\.→ see findings above.serialport::new→ every.timeout(Duration::…)call has an explicit per-read cap (2s reset, 3s native verify, 10s native write, 50ms / 100ms probe paths). Per-read caps are correct; the missing piece is wall-clock caps on the surrounding library calls.loop {/while→ all such loops in this crate carry anInstant::now() < deadlineorattempt < total_attemptsbound:teensy/port_discovery.rs:101,teensy/halfkay_probe.rs:45,teensy/first_byte_probe.rs:60,teensy/flash.rs:116,lib.rs:123. OK.thread::sleep/tokio::time::sleep→ all bounded by an enclosingwhile deadlineor a single fixed-duration backoff after a bounded loop.reqwest::→ no matches. This crate does not do HTTP.tokio::process::/\.await→ no matches (the deploy paths are sync; the daemon handler does thespawn_blocking).\.lock\(\)/RwLock→ only in#[cfg(test)]blocks (teensy/soft_reboot.rs:87TEST_ENV_LOCK,lib.rs:201,217,252,263ObservableDeployer). No production locks in deploy hot paths.post_deploy_recovery→ only the default impl inlib.rs:121and the test override; the documented 3s fast-poll is correctly bounded.
Out-of-scope notes
- Subprocess timeouts elsewhere:
fbuild_core::subprocess::run_commanditself implements the wall-clock cap viawait_timeout; that crate is being audited separately (thefbuild-corerow of audit: blocking operations with no timeout (meta) #802). - Daemon-side deploy handler: any
tokio::time::timeoutwrapping at the/api/deployHTTP boundary is incrates/fbuild-daemon/src/handlers/operations/deploy.rsand is covered by thefbuild-daemonaudit row of audit: blocking operations with no timeout (meta) #802. fbuild-serialpreemption protocol: WebSocket / IPC writes to fbuild-serial during deploy preemption are owned byfbuild-serialand are covered by that crate's audit row.reset.rs: usesserialport::new(...).timeout(Duration::from_secs(2))+ shortthread::sleep. Bounded, not a finding.size_check.rs/method_validation.rs: pure CPU / disk metadata, no blocking I/O.esp32/qemu.rs: builds argv only; never spawns QEMU itself (that's daemon-side).- espflash library version:
espflash = "4". If a future version exposes a configurablewall_clock_timeoutonFlasher, the suggestedtokio::time::timeoutwrappers can be replaced with a config knob. Until then the external wrapper is the only way.
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsTriage
Activity
fix(timeout): close #802 audit family — workspace-wide timeouts on bl…
zackees commentedon Jun 29, 2026
Closed by PR #843 (workspace-wide timeout sweep).