Repository navigation
Connection perf: multiplexed persistent streams over one tunnel (replace HTTP-per-call + pool of 50) #5756
Description
Multiplexed persistent streams over ONE tunnel for the sandbox↔laptop path (WS-C design, from #5701)
Part of epic #5412. Design evidence: WS-C connection-performance comment on #5701 (2026-11-04).
Measured problem (2026-11-04, trips.jsonl n=1149 + live probes)
- Warm single trivial call: client trip p50 369 ms (status-200, <2 s subset, n=800) while daemon work is 1–11 ms (daemon trips.jsonl; example trip aedc705d4294: client 445 ms vs daemon 1 ms). Transport ≈ 99% of cost.
- The path is HTTP/1.1-request-per-call over a pooled CONNECT tunnel (pool 50 in
fleet proxy serve). Even warm, one request/response costs ~170–600 ms over the runtime-proxy (:3130) + DERP hairpin path. A warm RTT is the floor for EVERY operation, and every CLI operation is a separate call. - Cold CONNECT measured 1,434 ms trip (same-minute warm trip: 212 ms) — CONNECT setup ≈ 1.2–1.8 s, and the ramp test on Incident record 2026-11-04: afternoon responsiveness loss — sandbox wedged after post-boot herd (daemon restart, 88-min sampler silence); laptop path exonerated 21:00–21:28 UTC #5701 showed CONNECT setup serializes under concurrency (128 tunnels = 14.6 s wall, no failures).
- Failure amplification: during proxy wobble, trips cluster at 502 after ~16.3 s (p50, n=119) and no-status after ~26 s (p50, n=73) — retry chains multiply the herd.
Proposal
Replace HTTP-per-call over N pooled tunnels with ONE (or two, for redundancy) persistent tunnel carrying multiplexed streams:
- Framed stream multiplexer on the tunnel (the broker already speaks length-prefixed frames on its unix socket — extend that framing end-to-end): each request is a stream (id, port/service, method, path, body) over a single TCP/TLS-less CONNECT tunnel; responses interleave by stream id. Candidates: yamux-style custom framing in the existing Rust
fleet-proxy, HTTP/2 over the tunnel (h2 gives streams + HPACK + PING keepalive for free), or QUIC if the runtime proxy path ever allows UDP (it does not today — CONNECT/TCP only, so QUIC is out until that changes). - No per-call CONNECT, no pool checkout, no HTTP/1.1 head-of-line blocking per request; pool of 50 tunnels collapses to 1–2 tunnels + stream concurrency (ramp proved the tunnel path itself survives 128 concurrent; streams remove the serialized CONNECT setup that made the ramp slow).
- Keepalive: application-level PING on the tunnel (replaces per-pool-connection health keepalives; current Python-broker constants KEEPALIVE_S=180 / idle-exit 3600 s carry over as starting values, but the broker must not idle-exit while a tunnel with in-flight streams is healthy — exit only when the tunnel is dead AND no calls for 3600 s).
- Backpressure + bounded retry: per-stream timeout with a hard ceiling well under the current 16–26 s failure cluster; one retry on a FRESH stream (not a fresh CONNECT) when the tunnel is alive; tunnel re-establishment is single-flight (one CONNECT at a time, queued streams wait on it) so a proxy wobble cannot stampede 50 CONNECTs.
- TCP_NODELAY on tunnel sockets end-to-end (absent in the Python stack today: grep finds no TCP_NODELAY in laptop-agent.py / laptop-agent-broker.py / laptop-agent-daemon.py) and coalesce header+body into one write per frame to avoid Nagle/delayed-ACK stalls on small request/response pairs.
Acceptance
- Warm trivial exec trip p50 ≤ 50 ms (from 369 ms) at comparable load, daemon duration unchanged.
- 128 concurrent requests over the single tunnel: 0 failures, wall ≤ ramp's 128-tunnel wall (14.6 s) by ≥5×.
- Failure injection (kill runtime proxy briefly): calls fail or recover in ≤2 s, no 16–26 s cluster in trips.jsonl.
- Trip logs on both sides keep the same trip id so the client/daemon split stays measurable.
Risks: single tunnel = single point of congestion/failure → mitigate with 2 tunnels + stream load-balancing and the single-flight reconnect above. HTTP/2 dependency in Rust (h2 crate) vs hand-rolled framing is an implementation choice for the builder; the wire contract (streamed frames, per-stream ids) is the requirement.
Activity
developing-today commentedon Nov 5, 2026
CODE-1443
dezren39 commentedon Nov 5, 2026
🤖 Agent note · handed off
developing-today commentedon Nov 5, 2026
🤖 Agent note · handed off
developing-today commentedon Nov 5, 2026
🤖 Agent note · handed off
dezren39 commentedon Nov 5, 2026
🤖 Agent note · picked up
dezren39 commentedon Nov 5, 2026
🤖 Agent note · triage complete
Triage at current main (2026-11-05, 97aecf1):
TunnelPool, and fleet-proxy work found no relevant open PR. Parent incident/design issue Incident record 2026-11-04: afternoon responsiveness loss — sandbox wedged after post-boot herd (daemon restart, 88-min sampler silence); laptop path exonerated 21:00–21:28 UTC #5701 is CLOSED, but its stage-2 note explicitly records the measured 50/64/128 ramp (zero failures through 128; 14.6 s at 128) and files Connection perf: multiplexed persistent streams over one tunnel (replace HTTP-per-call + pool of 50) #5756/Connection perf: coalesce queue claim flow to 1 round trip, cross-service batch, real file-transfer verb #5757/Session proxy: fail fast (≤2s, not 16-26s), single-flight recovery, timestamped broker log, pool stays 50 #5758 as distinct follow-ups. No hard-stop labels were present before this claim.97aecf1. Current broker code keeps a per-portTunnelPool(default 50) ofHttpConns inpkgs/fleet/crates/fleet-proxy/src/serve.rs;pkgs/fleet/crates/fleet-proxy/src/http.rsdoes one HTTP/1.1 request/response exchange on each connection;pkgs/fleet/crates/fleet-proxy/src/tunnel.rsestablishes the CONNECT tunnel and enables TCP_NODELAY. There are no stream IDs or response demultiplexing in this path. The existing pool is therefore not the requested single/two-tunnel multiplexed transport.Ordering / routing: #5756 remains the owner for the multiplexed-transport design and acceptance; no separate duplicate queue task was created. Keep round-trip coalescing (#5757) and fail-fast/recovery policy (#5758) as sibling scopes; neither is a blocker to #5756. No code changed in this triage.