design: restore broker-mediated serial proxy + finish running-process broker (port custody, disconnect events, multi-client singleton, cross-version device lease) #592

TL;DR

The original architecture had fbuild as the single owner of every serial port, with clients receiving thin proxy handles through the daemon broker. That model has eroded: FastLED's RpcClient now opens raw fbuild.api.SerialMonitor instances per logical session and physically tears down / re-opens the OS port on every error path. The running-process broker that was supposed to coordinate this is wired in code (fbuild daemon running-process reports Selected path: broker, Mode: direct-fallback) but its service definition file is missing on disk, so every fbuild invocation silently falls back to direct mode.

This issue is a request to re-land the broker as the canonical serial-port owner, redesign the proxy lifecycle so clients cannot cause physical teardown, add explicit disconnect / re-enumeration events, and finish the cross-version device-lease story so questions like "who currently has the ESP32-S3 on COM22?" have a single, authoritative answer.

1. Today's symptom (caught live, 2026-06-14)

Live repro on Windows 10 with a Teensy 4.0 (COM21, VID 16C0:0489), two ESP32-S3 dev boards (COM9, COM22 — both 303A:1001), and an LPC845-BRK (COM10).

> bash autoresearch teensy40 --object-fled -p COM21 --skip-lint --timeout 180
deploy succeeded (full flash)
Deploy succeeded (fbuild) [15.8s]
WARN: Port COM21 not available after 3s, proceeding anyway...
WARN: PIN DISCOVERY ERROR
   Serial error: WebSocket error: IO error: An existing connection was forcibly closed
   by the remote host. (os error 10054)
   Falling back to default pins
ERR:  GPIO PRE-TEST ERROR
   Serial connection error: cannot schedule new futures after shutdown
AUTORESEARCH ABORTED — GPIO PRE-TEST FAILED

What actually happened:

1. fbuild flashed the Teensy successfully in 15.8 s.
2. The Teensy USB stack re-enumerated post-flash. The COM port number moved (COM21 -> COM20) and the PID changed (16C0:0489 -> 16C0:0483).
3. The FastLED test framework called RpcClient.close() on a ping-failure recovery branch (ci/autoresearch/gpio.py:70), which calls FbuildSerialAdapter.close(), which calls self._executor.shutdown(wait=False) — the per-adapter thread executor is permanently dead from this point (ci/util/serial_interface.py:177).
4. The next reconnect attempt tries to submit work to the dead executor → cannot schedule new futures after shutdown.
5. Meanwhile the WebSocket to fbuild-daemon had already been forcibly closed because the underlying SerialMonitor's port handle went stale.

Direct-pyserial reconnect to the new COM20 worked instantly — confirming the firmware and driver were fine; only the proxy lifecycle was broken.

ci/autoresearch/driver_sweep.py:214 already documents the workaround the FastLED side adopted:

# Connect to device using pyserial (avoids fbuild daemon port locking issues)
iface = create_serial_interface(port, use_pyserial=True)

So both halves of the seam are visible from the FastLED tree: (a) the close-and-reopen pattern, and (b) a use_pyserial=True escape hatch that bypasses the broker entirely. The broker is paying the cost of being on the critical path without delivering the benefits it was designed for.

2. Where the broker currently is

fbuild daemon running-process (today, this machine):

Service:              fbuild
Isolation (local):    SHARED_BROKER
Isolation (CI):       EXPLICIT_INSTANCE "ci-trusted"
Min version:          1.0.0
Payload protocol:     0x7EB1
fbuild proto version: 1
Encoding lane:        json-direct + prost-broker (parity-tested)
Service definition:   C:\Users\niteris\AppData\Roaming\running-process\services\fbuild.servicedef
Selected path:        broker
Mode:                 direct-fallback
Daemon endpoint:      http://127.0.0.1:8765
Daemon binary:        ...\.venv\Scripts\fbuild-daemon.exe
Daemon binary exists: yes

%APPDATA%\running-process\services\ contains zccache.servicedef but no fbuild.servicedef — so Selected path: broker is aspirational and Mode: direct-fallback is operational. Every client lands on the direct path, where the close-and-reopen seam lives.

fbuild device list already enumerates VID/PID per port, fbuild device lease/release/take already exist, and fbuild daemon locks already prints Port Locks and Project Locks headings. The bones of the broker are in the binary — it is just not switched on, and FastLED is not using it.

3. Design — restore the broker as the canonical port owner

3.1 Proxy lifecycle (correct shape)

        +---- client A (autoresearch GPIO test) ----+
        |     SerialProxy.write / events            |
        v                                           |
  +----------------------------------------------+  |
  |            fbuild-daemon broker              |<-+
  |  +--------------------------------------+    |  |
  |  |  PortSession(COM21, lease=exclusive) |    |  |
  |  |  - single OS handle                  |    |  |
  |  |  - ring-buffer of RX bytes           |    |  |
  |  |  - subscriber list [A, B, C]         |    |  |
  |  |  - re-enum watcher                   |    |  |
  |  +--------------------------------------+    |  |
  +----------------------------------------------+  |
        ^                                           |
        +---- client B (`fbuild monitor COM21`) ----+

Concretely:

• Daemon owns every COM handle. pyserial.Serial(port, ...) is called exactly once per port per epoch, inside the daemon.
• Clients hold opaque proxy handles (SerialProxy) — a WebSocket frame channel keyed by (port, session_id, subscriber_id). The proxy has no OS resources.
• Refcount on the underlying port: opens on first subscriber, closes on last subscriber leaving. subscriber_id is a UUID so that subscriber leaks become diagnostic (fbuild daemon locks lists stale subscribers with last-heartbeat).
• Logical close vs physical close: SerialProxy.close() is logical — it removes the subscriber from the port's subscriber list and returns. It does not touch the OS handle. The daemon decides when to close the handle.
• Writes are arbitrated: only the holder of an exclusive lease may write; monitor leases are read-only. Concurrent exclusive writes are serialized at the daemon, not raced at the OS layer.

3.2 Teardown of the source connection

Two events trigger source-side teardown:

1. Subscriber refcount hits zero. The daemon waits a configurable grace window (default 2 s) — this absorbs the client.close() -> new RpcClient(...).connect() close/reopen pattern that is currently in gpio.py without thrashing the port. If a new subscriber arrives in the grace window, the port stays open and the new subscriber resumes with the daemon's RX ring-buffer playback (so it does not miss boot output).
2. OS reports the device is gone (Windows ERROR_FILE_NOT_FOUND/ERROR_DEVICE_NOT_CONNECTED, POSIX ENXIO/ENODEV). The daemon closes the handle, marks the session DETACHED, and emits a port_disconnected event to every subscriber. Subscribers do not tear down — they wait for a port_reattached event (see 3.3).

In both cases the daemon emits structured events on the WebSocket so the client side never needs to guess whether the port is alive.

3.3 Disconnect / re-enumeration detection

The daemon runs a single OS-level watcher (Windows: RegisterDeviceNotificationW on GUID_DEVINTERFACE_USB_DEVICE or WM_DEVICECHANGE; Linux: udev netlink; macOS: IOKit IONotificationPort) and emits the following events to subscribers:

{"event": "port_disconnected", "port": "COM21", "reason": "usb_disconnect",
 "vid_pid": "16C0:0489", "last_rx_ts": "..."}

{"event": "port_renumbered", "port": "COM21", "new_port": "COM20",
 "reason": "vid_pid_change", "old_vid_pid": "16C0:0489", "new_vid_pid": "16C0:0483",
 "serial": "15821020"}

{"event": "port_reattached", "port": "COM20", "previous_port": "COM21"}

The session identity follows the device serial number, not the COM number. A subscriber that opened (port="COM21") and survives a re-enumeration gets a port_renumbered event and the daemon transparently rebinds the session to COM20. The teensy_loader-flash -> re-enum case (the one that broke today) becomes a non-event for the client.

Optional knob: device.lease(port="COM21", track_serial=True) — when True, the lease follows the serial number across renumeration. Default False for boards where the serial is not unique (cheap CH340-clone clones share serials).

3.4 Multiple fbuilds -> one daemon -> one underlying connection

This is the singleton-per-port invariant the broker is supposed to enforce:

• Every fbuild monitor, fbuild deploy --monitor, fbuild reset, bash autoresearch ..., and any Python script calling fbuild.api.SerialMonitor(port=...) enters through the broker. No direct pyserial.Serial(port=...) call remains in the codebase outside the daemon's own port owner.
• The broker enforces that there is exactly one PortSession per (host, port) tuple, regardless of how many client processes / fbuild versions / Python scripts connect.
• The fbuild daemon binary chosen at runtime is the highest-version binary registered for the fbuild service (see 4 for how the running-process service tracker picks). Lower-version client libraries connect through it via the negotiated protocol version (Payload protocol: 0x7EB1, fbuild proto version: 1).

Concrete flow with two clients:

$ fbuild monitor COM22                    # client A — version 1.12.4
... opens WebSocket; broker:
    PortSession(COM22).subscribers = [A]; OS handle opened.

$ bash autoresearch esp32s3 --rmt          # client B — version 1.12.5
... opens WebSocket; broker:
    PortSession(COM22).subscribers = [A, B];
    new RX bytes are fanned to both;
    write requests from A queued behind write requests from B (per-lease arbitration).

$ fbuild monitor COM22 --leave             # client A logical close
    PortSession(COM22).subscribers = [B];
    OS handle stays open.

# A SIGKILL'd?
    Heartbeat timer expires; subscriber A removed; same as logical close.

The cannot schedule new futures after shutdown failure mode disappears because no client owns the executor that talks to the OS — the daemon does.

3.5 Closing the seam in FastLED

Once 3.1–3.4 are in place, the FastLED-side fix is small and mechanical:

• ci/util/serial_interface.py::FbuildSerialAdapter.close() removes the self._executor.shutdown(wait=False) line. Close becomes idempotent and reusable.
• ci/autoresearch/gpio.py:70-95 (and three more sites at gpio.py:268-307, decode.py:263, ble.py:245) stop creating a fresh RpcClient on error recovery. They issue a reset_device() on the existing proxy and resume — the daemon's RX ring-buffer makes this safe.
• ci/autoresearch/driver_sweep.py:214-216 drops the use_pyserial=True escape hatch and the explanatory comment (avoids fbuild daemon port locking issues).

Happy to send the FastLED PR once the daemon-side semantics are nailed down.

4. Cross-version resource holder + "who has the ESP32-S3?"

The running-process service tracker (already used by zccache) is the right substrate. The fbuild service is wired in code; what is missing is the manifest file and the lessee-attribution columns.

4.1 Install fbuild.servicedef

The fbuild package install step should write %APPDATA%/running-process/services/fbuild.servicedef (and the POSIX equivalent under $XDG_CONFIG_HOME/running-process/services/). Mirroring the zccache.servicedef shape that is already on disk:

service:        fbuild
binary:         <abs path to fbuild-daemon.exe / fbuild-daemon>
manifest_dir:   <abs path to %APPDATA%/running-process/manifests>
version:        <this fbuild version>
min_version:    1.0.0
attrs:          package=fbuild
                running-process-tracker=FastLED/running-process#435
                consumer=fbuild
                vendor=FastLED

When multiple fbuild versions are installed across virtualenvs / system Python / CI containers, the broker picks the highest version whose min_version is satisfied by all live consumers. Older clients can still talk to a newer daemon through the protocol-version negotiation that is already in place (fbuild proto version: 1, Payload protocol: 0x7EB1).

4.2 Device lease as the answer to "who has it?"

fbuild device lease COM22 -t exclusive -d 'autoresearch esp32s3 --rmt' already exists; it just needs to be:

• Required for any write-path operation (deploy, reset, write-to-port). Deploy auto-acquires; subscribers without a lease cannot write.
• Surfaced on fbuild device list / status:

> fbuild device list --leases
PORT   DEVICE ID    HOLDER                                     LEASE      SINCE      DESCRIPTION
----------------------------------------------------------------------------------------------------------
COM9   303a:1001    (none)                                     —          —          USB Serial Device
COM20  16c0:0483    pid 23120 niteris@WS bash autoresearch...  exclusive  14:23 UTC  Teensy 4.0 (renum from COM21, serial 15821020)
COM22  303a:1001    pid 18044 alice@WS  fbuild monitor         monitor    14:18 UTC  ESP32-S3 dev board (DevKitC-1)

> fbuild device status COM22 --json
{
  "port": "COM22",
  "vid_pid": "303a:1001",
  "serial": "D8:3B:DA:41:64:C0",
  "session_open": true,
  "subscribers": [
    { "id": "a3e9...", "pid": 18044, "user": "alice", "cwd": "C:/work/fastled",
      "started": "2026-06-14T14:18:02Z", "lease": "monitor", "description": "fbuild monitor" }
  ],
  "writers": [],
  "rx_buffer_bytes": 8472,
  "last_rx_ts": "2026-06-14T14:23:51.014Z"
}

That answers "who has the ESP32-S3 on COM22, and what are they doing?" in one call. CI dashboards can poll device list --leases --json to show the device farm at a glance.

4.3 Conflict and contention UX

fbuild deploy -e esp32s3 -p COM22 when COM22 is held by another exclusive lease should fail fast with a structured error:

error: COM22 (ESP32-S3, 303a:1001) is exclusively leased
  holder:      pid 18044 (alice@WS), since 14:18 UTC
  description: "fbuild monitor"
  cwd:         C:/work/fastled
hint: another build is running. Wait for it to finish, or take ownership with:
      fbuild device take COM22 --reason "deploy for autoresearch"

device take already exists; it just needs to publish a structured lease_taken event so the displaced client can clean up gracefully instead of erroring on the next read.

5. Suggested rollout

Smallest viable PRs in order:

1. #0 — drop FbuildSerialAdapter._executor.shutdown(wait=False) in FastLED (ci/util/serial_interface.py:177). One-line bug fix; lets the existing close-and-reopen seam at least work until the broker takes over.
2. Esp32-s3 PSRAM use #1 — install fbuild.servicedef at package install time and flip Mode: direct-fallback -> Mode: broker when the manifest is present. Add a smoke test that asserts Selected path: broker, Mode: broker after install.
3. cdc on boot warning #2 — singleton PortSession in the daemon + WebSocket fan-out + grace-window close (3.1, 3.2).
4. Fix ESP32-S3 PSRAM board configuration in board JSON database #3 — disconnect / renumeration events with track_serial=True (3.3).
5. Add CDC on boot warning to ESP32 builds #4 — lease-attribution columns + structured contention errors (4.2, 4.3).
6. Add VSCode extension scaffold (vscode-fbuild/) #5 — FastLED-side cleanup PR removing use_pyserial=True workarounds and the per-error close-and-reopen sites (3.5).

Each step is independently shippable and leaves the system in a strictly better state.

6. Repro snippets / pointers

• Reproducer for the seam: bash autoresearch teensy40 --object-fled -p COM21 against a Teensy 4.0 that re-enumerates post-flash (any teensy_loader-flashed board does this). The error chain WebSocket forcibly closed -> cannot schedule new futures after shutdown is deterministic on Windows.
• FastLED workaround already in the tree (proves the issue is felt downstream): ci/autoresearch/driver_sweep.py:214 — # Connect to device using pyserial (avoids fbuild daemon port locking issues).
• Broker state inspection: fbuild daemon running-process (shows Mode: direct-fallback because fbuild.servicedef is absent), fbuild daemon locks (shows the singleton-port headings already there), fbuild device list / lease / take (the lease CLI is built; the lessee-attribution columns are not).
• Servicedef reference: %APPDATA%\running-process\services\zccache.servicedef is the shape to mirror for fbuild.servicedef.

Filed at the user's request after losing a debugging session to the close-and-reopen seam this afternoon. Happy to send any of the PRs above, particularly #0 and #5 on the FastLED side.