audit: blocking operations with no timeout in fbuild-build (sub-issue of #802) #809

Intro

Audited crates/fbuild-build/ for blocking operations that have no
timeout / deadline / cancellation. Scope: build orchestration, per-platform
orchestrators (AVR, ESP32, ESP8266, RP2040, STM32, Teensy, SAM, NRF52, CH32V,
Renesas, Silabs, NXP-LPC, Apollo3, generic-arm), the shared compile pipeline,
linker invocations, the embedded zccache compile dispatch, the extra_scripts
Python harness, and the symbol/bloat analyzers (nm, objdump, c++filt,
size, objcopy, ar).

The dominant failure mode in this crate is subprocess invocations of toolchain
binaries via fbuild_core::subprocess::run_command(..., timeout: None).
That helper does support a timeout: Option<Duration> (with Timeout error
mapping and process.kill() on expiry — see crates/fbuild-core/src/subprocess.rs
lines 81-208), so the fix at every site below is mechanical: pass
Some(Duration::from_secs(N)) with an N chosen to fit the operation
(5-30s for utility tools like nm/size/objcopy/-dumpversion, longer for
actual link steps that legitimately run for minutes on large ESP32 firmware).

A handful of utility sites already do this correctly (e.g. esp32_linker.rs:381
elf2image with 30s, boot_artifacts.rs:94/145, shrink/probe.rs:352,
esp8266_linker.rs:262) — they should be the template for the missing ones.

The second class of finding is the embedded zccache dispatch in
zccache_embedded.rs / compile_backend.rs. Every per-translation-unit compile
flows through FbuildZccacheService::compile_blocking, which calls
tokio::runtime::Handle::block_on(svc.compile(req).await) with no
tokio::time::timeout(...) wrapper. If the upstream ZccacheService::compile
ever hangs (network-mode cache backends, deadlocked fingerprint scan, etc.)
the build worker thread parks forever. This is the single highest-impact
finding because it is on the hot compile path inside the daemon.

The third class is the extra_scripts Python harness invocation in
script_runtime.rs. User-supplied extra_scripts = [...] in platformio.ini
are evaluated by spawning Python with no timeout — a hostile or buggy script
can wedge the daemon's build pipeline indefinitely.

Findings

What was searched

Patterns run via Grep over crates/fbuild-build/src/:

• Command::new — no production matches (only test files in crates/fbuild-build/tests/)
• \.output\(\) — only tests
• \.wait\(\)|\.wait_with_output\(\)|wait_timeout — only Barrier::wait() in tests
• reqwest|TcpStream|TcpListener — no matches in src/ (only Cargo.toml dep)
• tokio::time::timeout|tokio::time::sleep|tokio::process — no matches in this crate
• \.await — only in compile_backend.rs and zccache_embedded.rs (audited individually above)
• \.lock\(\) — all are std::sync::Mutex::lock(), scoped to local data structures (work_iter, first_error, results_slot, build_log, COMPILER_IDENTITY_CACHE, fingerprint cache entries), not external state. No deadlock risk identified — locks are held only across small map/insert/push ops, never across await points or subprocess waits.
• \.read\(\)\.unwrap|\.write\(\)\.unwrap — no matches (no RwLock use)
• \.recv\(\)|recv_timeout|mpsc:: — only mpsc::Sender in build_output.rs and lib.rs (no receivers in this crate; receiver lives in fbuild-daemon's websocket pump and is out of scope)
• thread::sleep|Instant::now|Duration::from — Instant::now is everywhere for perf timing (not for polling). thread::sleep only in test code.
• block_on — single production occurrence in zccache_embedded.rs:216 (covered above).
• run_command\( / run_command_with_stdin / run_command_passthrough — every call site catalogued. Sites with Some(Duration::...) already (good template):
  • esp32_linker.rs:381 — 30s for esptool elf2image
  • esp8266_linker.rs:262 — 30s for esptool elf2image
  • esp32/orchestrator/boot_artifacts.rs:94 — 30s for bootloader elf2image
  • esp32/orchestrator/boot_artifacts.rs:145 — 10s for gen_esp32part.py
  • shrink/probe.rs:352 — 5s for probe compile

Files spot-checked for full context:

• crates/fbuild-core/src/subprocess.rs — confirmed run_command honors timeout via process.wait(timeout) and kills the child on ProcessError::Timeout.
• crates/fbuild-build/src/compiler.rs — confirmed compile_source is the single funnel from every platform compiler module into compile_blocking.
• crates/fbuild-build/src/parallel.rs — confirmed the loop {} in the worker terminates when the work_iter runs out; not a busy-poll.
• crates/fbuild-build/src/build_fingerprint/fast_path.rs — Mutex use is bounded to local map operations.

Out-of-scope notes

• crates/fbuild-build/tests/*.rs and *_tests.rs modules — test code with Command::new, Barrier::wait(), and thread::sleep for synchronization. These can hang CI if they break but the fix is per-test (most already use bounded sleeps under 50ms).
• Drop impls — no Drop impls in this crate perform blocking IO. (The embedded ZccacheService Drop is upstream in zccache and out of scope per audit: blocking operations with no timeout (meta) #802's audit guidance.)
• running_process crate internals — the subprocess primitive itself has timeout support; bugs in its implementation are out of scope for this audit.
• fbuild-daemon HTTP handlers — out of scope (separate auditor area).
• compile_database/ — pure JSON serialization; no blocking IO besides std::fs which is filesystem-bounded.
• eh_frame_policy*.rs, flag_overlay.rs, arduino_props.rs, package_override.rs, framework_libs.rs, framework_core_cache.rs, resolution.rs, lib.rs — pure config / data transformation, no subprocess or locking concerns.
• source_scanner/ — uses io::stderr().lock() for line-by-line output flushing; not unbounded blocking.

Sub-issue of #802.