design: comprehensive Teensy deployer + monitor (replaces #432; closes multi-failure-mode pain cluster from FastLED#2628 hardware bring-up) #433

Goal

Make fbuild deploy -e teensyXX and fbuild monitor -p <port> (or --monitor on deploy) work end-to-end for Teensy 3.x/4.x on a wedged Windows host without manual button presses, without USB cable reseats, without the PlatformIO compatibility fallback. Today the failure modes documented in #432 force users into fbuild deploy --platformio which defeats the point of having a native fbuild path. This issue replaces #432 with a wider scope: the deployer, the serial-port hygiene around it, and the post-deploy monitor as one cohesive subsystem.

Session that motivated this issue: FastLED/FastLED#2628 SIMD verification on Teensy 4.0 — the build, flash, boot, RPC-monitor pipeline failed at every stage in ways that pure code can't paper over.

What works today (after #430/#431)

• TeensyDeployer in crates/fbuild-deploy/src/teensy.rs is wired into the daemon's Platform::Teensy arm.
• find_teensy_loader_cli() discovers the bundled tool at ~/.platformio/packages/tool-teensy/teensy_loader_cli{.exe} and falls back to PATH.
• The TeensyDeployer invokes teensy_loader_cli --mcu=TEENSYxx -w -v firmware.hex with a 60s timeout.

What does NOT work — failure modes observed in one session

Observed on a Teensy 4.0 / Windows 10 host. Each is a separate bug surface, but they cluster — the deployer needs to treat them as one state machine.

1. No soft-reboot trigger (issue #432, restated)

teensy_loader_cli on Windows prints Soft reboot is not implemented for Win32 and waits for the program button. After the first flash, the freshly-flashed user firmware is running — the next fbuild deploy cannot put it back into HalfKay without manual intervention. CI/iteration is dead.

Fix: Open the device's CDC ACM port at baud 134 before invoking teensy_loader_cli. The Teensyduino USB stack treats this as a signal to drop into HalfKay. ~250 ms later the device re-enumerates as VID_16C0&PID_0478 (HalfKay HID), and the loader runs unattended.

2. Intermittent USB write failures during programming

Observed 6/10 times across one session: teensy_loader_cli finds HalfKay, starts Programming., then errors out at byte 1 with error writing to Teensy. No retry, no diagnostic, no recovery. On retry it sometimes works after several attempts; sometimes a USB cable reseat is required.

This is consistent with a USB host-controller / driver quirk on Windows where the WinUSB / HID transfer layer occasionally rejects the first OUT after enumeration. The Teensyduino people are aware (see hint output (hint: press the reset button)); the loader just bails.

Fix: Bounded retry loop in TeensyDeployer::deploy — re-invoke teensy_loader_cli up to N times (default 5) with a short backoff between attempts. Each attempt is a fresh subprocess. Stop on first success. Surface a structured diagnostic if all N fail (USB device state at each attempt, USB hub topology, last stderr) so the user knows whether the next step is "wait", "reseat cable", or "try another port".

3. Stale firmware.hex confusion (deploy uses the wrong file)

teensy_loader_cli flashed an old Blink firmware when AutoResearch was the requested example, because fbuild deploy resolves the path to whatever was last built at .build/pio/teensyXX/.fbuild/build/teensyXX/release/firmware.hex without re-running the build for the requested example. The user has to either pre-fbuild build --examples X or accept the wrong firmware silently.

Fix: fbuild deploy -e teensy40 --examples Foo (parallel to fbuild compile) should re-run the example build first, then flash the just-produced HEX. Document the path the deployer reads from and emit it in verbose mode so mismatches are obvious. Optionally: include the example name in the HEX path so different examples produce different artifact paths.

4. Post-deploy port enumeration race

After flash, Teensy 4 takes ~1-2 s to USB-re-enumerate as a CDC ACM device. If a monitor (or RPC client) opens the port immediately after deploy returns, the port either doesn't exist yet or the firmware's setup-time Serial.begin hasn't completed. Result: zero bytes drained, monitor declares timeout.

Fix: TeensyDeployer::deploy returns the post-flash port name in DeploymentResult.port (which is None today for Teensy). The deployer waits for the re-enumeration before returning success:

• After teensy_loader_cli prints Booting, poll serialport::available_ports() for up to 5 s looking for a CDC ACM at VID_16C0&PID_048x that wasn't present before.
• Return that port name; the post-deploy monitor opens against it.

5. Firmware-silent failure modes are indistinguishable from deploy success

If teensy_loader_cli reports Booting but the firmware crashes in setup() before its first Serial.print, the user sees: fbuild deploy: SUCCESS, port enumerates, but fbuild monitor drains zero bytes forever. Today there is no way to tell whether the flash failed silently, the firmware is hung, or the monitor's pattern is wrong.

Fix: Default post-deploy "did setup print anything in N seconds?" probe.

• Deploy returns. If --monitor was passed or the daemon's deploy handler is in attached mode, immediately attach a probe at the post-flash port.
• Probe waits up to --first-byte-timeout=10s for ANY byte at the configured baud (default 115200). If nothing arrives, emit a structured diagnostic: "Teensy enumerated as but produced zero serial bytes in 10s. The firmware may be hanging in setup(). Try: (a) press the program button to re-enter HalfKay and reflash; (b) check the build's usb_type (USB_SERIAL vs USB_MIDI_SERIAL); (c) verify your sketch's Serial.begin(115200) runs to completion."
• This is purely advisory — does not gate deploy success, but surfaces a clear next step.

6. Timeout defaults too tight for first-flash scenarios

TeensyLoaderParams::timeout_secs = 60 is fine for the unattended case (with magic-baud reboot, flash completes in ~3 s), but fails the first-flash case where the user has to press the program button — which can take 30+ seconds for someone walking up to a board they have not touched. The -w (wait-for-device) flag holds the loader open indefinitely, but our subprocess timeout kills it at 60 s.

Fix: Two-tier timeout:

• flash_timeout_secs (default 30) — once HalfKay is detected and programming starts, hard cap on the actual flash (currently ~5 s for a Teensy 4 with 261 KB).
• wait_for_halfkay_timeout_secs (default 180) — how long to wait for HalfKay to appear before programming starts. After magic-baud trigger this should resolve in ~1 s; the 180 s budget is the "user pressing the program button" case.
• The two together never wedge a subprocess past the sum.

Monitor side (the second half of "deployer and monitor")

The existing handlers/operations/monitor.rs is platform-agnostic — it just opens a serialport and runs the halt/expect state machine. For Teensy specifically the gaps are:

7. USB CDC reconnection across deploy

After fbuild deploy --monitor flashes a new firmware, the port handle that the monitor would attach to is the pre-flash port — which dies during enumeration and a new one appears post-flash. If the user wrote --port COM21 in the CLI, COM21 might literally not be the same device after deploy.

Fix: The deploy handler passes the post-flash port (item #4 above) to the monitor. CLI users get a clear log line: Teensy re-enumerated as COMxx after flash (was COMyy). Monitor attaching to COMxx.

8. Teensy usb_type mismatch with monitor baud rate

Teensy USB Serial is CDC ACM — baud rate is ignored on the device side, but Windows still requires a baud to be set on the host. If usb_type was built as USB_MIDI_SERIAL or USB_RAWHID, there is no Serial endpoint at all, and the monitor will open a port that produces zero bytes (similar symptom to #5).

Fix: Read usb_type from the build's environment (saved at build time alongside firmware.hex). If it is USB_MIDI_SERIAL, emit a warning that the monitor is targeting the Serial endpoint, not MIDI. If it is USB_RAWHID, emit an error — fbuild monitor does not speak HID.

9. Crash-handler integration

Teensy 4 has built-in CrashReport that prints prior-crash info on next boot if Serial is available. A robust monitor should:

• Detect the CrashReport: prefix in the first few lines after enumeration
• Treat it as a notable event (do not gate success, but surface it loudly)
• Optionally save the crash report to a session log file so iterations have a forensic trail

Proposed module layout

crates/fbuild-deploy/src/teensy/
├── mod.rs              -- pub use TeensyDeployer
├── deployer.rs         -- TeensyDeployer impl (orchestrates the state machine below)
├── soft_reboot.rs      -- baud-134 trigger; returns "device entered HalfKay" or error
├── halfkay_probe.rs    -- USB enumeration probe for VID_16C0&PID_0478
├── flash.rs            -- thin wrapper around teensy_loader_cli with the retry loop
├── port_discovery.rs   -- post-flash CDC ACM enumeration + new-port detection
├── first_byte_probe.rs -- "firmware actually produces output" probe (item #5)
└── usb_type.rs         -- read usb_type from build artifact, advise the monitor

fbuild-deploy already pulls in serialport; everything above can be implemented without new dependencies.

The existing Deployer trait stays as-is; TeensyDeployer::deploy becomes the state machine that calls the modules above in sequence.

State machine

                   ┌────────────────────────────┐
                   │ TeensyDeployer::deploy()   │
                   └────────────┬───────────────┘
                                │
                  ┌─────────────▼─────────────┐
                  │  Pre-flash port snapshot  │   serialport::available_ports()
                  └─────────────┬─────────────┘
                                │
                  ┌─────────────▼─────────────┐
                  │  Is CDC ACM at port?      │   port==Some && enumerable
                  └──────┬──────────────┬─────┘
                        yes             no
                         │              │
        ┌────────────────▼──┐      ┌────▼─────────────────────────┐
        │ soft_reboot::baud_│      │ halfkay_probe::wait_for_halfkay
        │ 134_trigger(port) │      │ (wait_for_halfkay_timeout_secs)
        └────────┬──────────┘      └────────────┬─────────────────┘
                 │                              │
                 └─────────────┬────────────────┘
                               │
                 ┌─────────────▼─────────────────┐
                 │ flash::run_with_retry(N=5)    │
                 │  - calls teensy_loader_cli    │
                 │  - bounded backoff on failure │
                 │  - succeeds on "Booting"      │
                 └─────────────┬─────────────────┘
                               │
                 ┌─────────────▼─────────────────┐
                 │ port_discovery::wait_for_new_ │
                 │ cdc_acm(pre_snapshot)         │   (up to 5 s)
                 └─────────────┬─────────────────┘
                               │
                 ┌─────────────▼─────────────────┐
                 │ first_byte_probe::optional(   │   (up to first_byte_timeout)
                 │   port, advisory_only=true)   │
                 └─────────────┬─────────────────┘
                               │
                 ┌─────────────▼─────────────────┐
                 │ return DeploymentResult {     │
                 │   port: Some(new_port),       │
                 │   message: "flashed; new port │
                 │     COMxx; first byte at Nms" │
                 │ }                             │
                 └───────────────────────────────┘

Failure at any stage returns a DeploymentResult { success: false, message: <stage-specific>, stderr: <last loader output> } — the daemon handler already propagates this to the CLI verbatim.

API additions

TeensyLoaderParams extended:

pub struct TeensyLoaderParams {
    pub wait_flag: String,            // existing
    pub verbose_flag: String,         // existing
    pub flash_timeout_secs: u64,      // NEW (default 30)
    pub wait_for_halfkay_timeout_secs: u64,  // NEW (default 180)
    pub flash_retries: u32,           // NEW (default 5)
    pub flash_retry_backoff_ms: u64,  // NEW (default 1500)
    pub first_byte_timeout_secs: u64, // NEW (default 10, 0 disables)
    pub baud_134_trigger: bool,       // NEW (default true; off for testing)
}

Env-overridable for unattended CI:

• FBUILD_TEENSY_FLASH_RETRIES
• FBUILD_TEENSY_FIRST_BYTE_TIMEOUT_SECS
• FBUILD_TEENSY_DISABLE_BAUD_134_TRIGGER (escape hatch)

DeploymentResult.port filled in for Teensy (post-flash CDC port).

Acceptance criteria

• fbuild deploy -e teensy40 succeeds on a Teensy with running user firmware, no button press, on Windows AND Linux/macOS.
• fbuild deploy -e teensy40 --monitor produces serial output within first_byte_timeout_secs for a sketch that calls Serial.begin(115200); Serial.println("ready");.
• Five consecutive fbuild deploy -e teensy40 cycles complete unattended (no human in the loop) on a single connected Teensy.
• --examples MySketch rebuilds and flashes the named example, not whatever happens to be at the build path.
• When the deployer fails (USB write error, no HalfKay, firmware silent), the user sees a structured diagnostic naming the failing stage and a concrete next-step suggestion. Not "deploy failed: error writing to Teensy".
• Cross-platform: tested on Windows 10/11, Ubuntu 22.04+, macOS 14+. Linux and macOS have -s working in teensy_loader_cli but the magic-baud trigger should be used uniformly so behaviour is identical across hosts.
• The retry loop emits one log line per attempt (attempt 2/5 failed: error writing to Teensy; backing off 1500ms).
• Existing TeensyDeployer unit tests still pass; new tests cover: baud-134 path (mocked serial), retry exhaustion path, port-rediscovery path, first-byte-probe path.

Phasing

Phase 1 (closes #432 directly):

• soft_reboot::baud_134_trigger() + integration in deploy()
• Pre-flash port snapshot + skip-trigger-when-already-HalfKay
• flash::run_with_retry (retry count + backoff)
• Better stderr/stdout surfacing in DeploymentResult

Phase 2:

• port_discovery::wait_for_new_cdc_acm() post-flash
• DeploymentResult.port filled in for Teensy
• Two-tier timeout (flash vs wait-for-halfkay)

Phase 3:

• first_byte_probe::optional()
• usb_type advisory
• CrashReport detection in monitor

Phase 4:

• --examples Foo chaining for fbuild deploy
• Build artifact path includes the example name

Each phase is independently mergeable, each phase has a verifiable smoke test on real hardware (one Teensy 4 attached), and each phase fixes a specific failure mode listed above.

Risks

• serialport baud-134 portability: needs verification that SerialPortBuilder::baud_rate(134) is honored on macOS + Linux + Windows. PJRC's reference impl uses raw SetCommState on Windows; if serialport does not, we may need a platform-specific branch.
• USB hub topology: the retry loop helps but does not fix bad cables / hubs. We should explicitly recommend direct-to-host USB in the diagnostic.
• Teensy usb_type=USB_RAWHID sketches: there is no Serial endpoint — monitor will always be silent. The advisory in item Fix ESP32-C2 build: support multiple archive formats in framework extraction #8 mitigates user confusion.
• HalfKay clock drift: a Teensy stuck in HalfKay for >10 minutes is reported (Teensyduino forums) to occasionally need a power-cycle. Out of scope; we surface a diagnostic when retries exhaust.

Out of scope

• JLink / non-USB upload paths (Teensy 4.1 has it, deferred until someone needs it).
• Teensy LC bootloader differences (it is a Cortex-M0 with a different HalfKay vid/pid — same architecture should apply but separate validation).
• Crash-stack symbolication (Teensyduino's CrashReport.h is already on-device; we just want to surface it on the host).

References

• feat(deploy): teensy soft-reboot trigger (magic baud 134) — close 'Soft reboot is not implemented for Win32' gap #432 — superseded by this issue (baud-134 trigger is Phase 1 here)
• feat(deploy): wire Teensy dispatch arm so 'fbuild deploy -e teensyXX' actually runs the implemented TeensyDeployer #430, feat(deploy): wire Teensy dispatch arm so 'fbuild deploy -e teensyXX' actually runs (#430) #431 — existing dispatch wire-up
• feat(simd): add Cortex-M DSP-extension SIMD backend for Teensy 3.x / 4.x (close the SIMD-dispatch gap) FastLED#2628 — the SIMD work that needs unattended Teensy deploy
• Teensyduino USB stack: ~/.platformio/packages/framework-arduinoteensy/cores/teensy4/usb_serial.c (baud-134 detection)
• PJRC reference: https://www.pjrc.com/teensy/loader.html

🤖 Generated with Claude Code