Add metadata-driven emulator runners for crash-oriented post-build testing #21

Goal

Add a reliable build -> emulator run -> crash/pass/fail classification path to fbuild so we can use the normal build phase to generate binaries, run them under emulators/simulators, and fail fast when the program crashes, times out, or never reaches a success marker.

This should be treated as a post-build test runner problem, not a hardware deploy problem.

Why this matters

fbuild already has most of the pieces needed for crash-oriented emulator testing:

• build orchestrators that emit real board artifacts
• daemon/CLI monitor semantics with --halt-on-success, --halt-on-error, --expect, and --timeout
• ESP32 crash decoding for Xtensa and RISC-V serial output
• imported board metadata that already knows about tools like simavr, qemu, and renode

What is missing is the emulator execution layer and the metadata plumbing to choose a runner.

Current repo state

Relevant code facts:

• crates/fbuild-cli/src/main.rs still rejects --qemu with QEMU deployment is not yet supported in the Rust port
• crates/fbuild-daemon/src/models.rs still exposes qemu and qemu_timeout
• crates/fbuild-daemon/src/handlers/operations.rs already has the monitor loop needed for timeout / expect / halt logic
• crates/fbuild-serial/src/crash_decoder.rs already decodes ESP32-family crash output
• imported board JSON files already advertise simulator/debug hints like simavr, qemu, and renode
• BoardConfig does not currently preserve enough of that metadata for runtime emulator selection

So the API shape already anticipates emulator execution. The missing part is implementation.

Recommended architecture

Do not bolt emulator support into hardware deployers.

Add a separate EmulatorRunner layer:

1. build using the normal orchestrator
2. collect an emulator-ready artifact bundle
3. select a runner from board metadata and target family
4. launch the runner
5. stream stdout/stderr into the existing monitor path
6. classify the result as Passed, Failed, Crashed, TimedOut, or Unsupported

This keeps emulator execution aligned with how simulator support is documented elsewhere and avoids mixing flashing logic with process execution.

Artifact model

Different emulators need different artifact shapes, so build results should expose a machine-readable bundle instead of a single firmware path.

Examples:

• SimavrRunner -> firmware.elf
• QemuEsp32Runner -> bootloader + partitions + app binary, or a merged image
• RenodeRunner -> firmware.elf plus board/machine script
• WasmRunner -> wasm/js output bundle

What we can realistically support first

Near-term

• AVR via simavr
• AVR via avr8js for JS-based/headless experimentation and possible browser/Node-backed smoke testing
• ESP32 Xtensa via QEMU, starting with esp32dev
• WASM as a simulation/smoke-test backend

Medium-term

• selected Renode-backed targets, once metadata-driven runner selection exists
• ESP32-C3 / other ESP RISC-V follow-ons after the Xtensa path is stable
• qemu-system-avr as a secondary AVR path once the abstraction exists
• rp2040js evaluation for RP2040 smoke-test coverage

Not near-term

• broad STM32 / nRF52 / Renesas emulation
• Raspberry Pi guest-OS style testing inside fbuild
• claiming that all boards with qemu / renode / simavr hints are supported
• timing-accurate hardware validation of LED/peripheral behavior

Concrete upstream repos relevant to this work

These are not generic categories; they are the actual technology families worth tracking because Velxio and the surrounding ecosystem use them directly:

Emulators and board/simulation components

• wokwi/avr8js
  • https://github.com/wokwi/avr8js
• wokwi/rp2040js
  • https://github.com/wokwi/rp2040js
• wokwi/wokwi-elements
  • https://github.com/wokwi/wokwi-elements
• wokwi/wokwi-features
  • https://github.com/wokwi/wokwi-features
• lcgamboa/qemu
  • https://github.com/lcgamboa/qemu
• davidmonterocrespo24/qemu-lcgamboa
  • https://github.com/davidmonterocrespo24/qemu-lcgamboa
• qemu/qemu
  • https://github.com/qemu/qemu

Tooling used around the emulator stack

• arduino/arduino-cli
  • https://github.com/arduino/arduino-cli
• earlephilhower/arduino-pico
  • https://github.com/earlephilhower/arduino-pico

The key point is that supporting �Velxio-style emulator coverage� actually spans multiple implementation families:

• JS/browser CPU emulators like avr8js and rp2040js
• MCU/system emulators like QEMU and QEMU forks
• machine/board/component layers like wokwi-elements
• normal compile toolchains that produce the binaries fed into those emulators

Near-future task list

Priority 0: metadata plumbing

• preserve simulator/debug-tool metadata from board JSON in BoardConfig
• define a board-facing emulator capability model instead of dropping those fields on import
• add a clear runtime selector for simavr, qemu, renode, avr8js, etc.

Priority 1: runner abstraction

• introduce an EmulatorRunner abstraction separate from hardware deploy
• add an emulator execution result type: Passed, Failed, Crashed, TimedOut, Unsupported
• make runner launch stream stdout/stderr into the existing monitor logic
• keep crash decoding reusable for emulator output when it matches serial output

Priority 2: command surface

• add fbuild test-emu or fbuild run-emulator
• reuse the normal build pipeline instead of inventing a second compile path
• allow --expect, --halt-on-success, --halt-on-error, and --timeout
• surface the full command line and captured output in failures

Priority 3: artifact bundles

• extend build results so emulator-capable targets expose all required files, not just one firmware path
• standardize artifact naming for ELF / HEX / BIN / bootloader / partitions / merged image
• add tests that assert emulator-ready artifact bundles exist for supported boards

Priority 4: first runnable backends

• implement SimavrRunner for Uno first
• extend AVR support to Mega and Leonardo
• implement QemuEsp32Runner for esp32dev
• add WASM smoke-test execution as a first-class runner

Priority 5: fixtures and CI

• add deterministic emulator fixtures that print success/failure markers
• add deliberate crash fixtures for ESP32
• add timeout fixtures to validate monitor behavior
• add golden tests for pass/fail/crash classification

Priority 6: scale-out path

• evaluate Renode as the medium-term path for broader MCU coverage
• add one metadata-driven Renode prototype for a known-good board
• evaluate whether avr8js and rp2040js fit as Node/headless runners, browser-backed tooling, or both

Recommended first deliverable

The shortest useful slice is:

• fbuild test-emu tests/platform/uno
• runner: simavr
• artifact: firmware.elf
• pass/fail: existing monitor pattern matching

Immediately after that:

• fbuild test-emu tests/platform/esp32dev
• runner: QEMU
• artifact bundle: bootloader + partitions + app binary or merged image
• result path: existing timeout and crash classification

Acceptance criteria

• board metadata exposes simulator/debug-tool hints needed for runtime selection
• a standalone EmulatorRunner path exists without going through hardware flashing logic
• fbuild test-emu works end-to-end for at least one AVR board
• fbuild test-emu works end-to-end for esp32dev with documented constraints
• emulator output participates in existing pass/fail/timeout/crash classification
• unsupported boards fail explicitly instead of pretending emulator support exists

References

• PlatformIO simulator overview: https://docs.platformio.org/en/stable/advanced/unit-testing/simulators/index.html
• PlatformIO QEMU docs: https://docs.platformio.org/en/stable/advanced/unit-testing/simulators/qemu.html
• PlatformIO Renode docs: https://docs.platformio.org/en/stable/advanced/unit-testing/simulators/renode.html
• PlatformIO SimAVR docs: https://docs.platformio.org/en/stable/advanced/unit-testing/simulators/simavr.html