Repository navigation
Session proxy: fail fast (≤2s, not 16-26s), single-flight recovery, timestamped broker log, pool stays 50 #5758
Description
Session-proxy: fail fast, size the pool from the ramp, timestamp the broker log (WS-C)
Part of epic #5412. Evidence: WS-C comment on #5701 (2026-11-04) + the #5701 stage-2 ramp.
Facts
- Ramp (stage 2, Incident record 2026-11-04: afternoon responsiveness loss — sandbox wedged after post-boot herd (daemon restart, 88-min sampler silence); laptop path exonerated 21:00–21:28 UTC #5701): 50/64/128 concurrent tunnels all succeeded (14.6 s wall at 128, latency ≈ serialized CONNECT setup). No ceiling ≤128 — so pool size is NOT a correctness limit today;
LAPTOP_AGENT_POOL=50(~/tools/init-ensure.sh) stays until the multiplexed-tunnel work (sibling issue) makes pool size moot. Do NOT raise to 128: more idle tunnels = more keepalive traffic and a bigger CONNECT stampede on proxy recovery, for zero measured benefit. - Failure mode is the real cost: in trips.jsonl (n=1149, 2026-11-04), status 502 trips have p50 16,294 ms (n=119) and no-status trips p50 25,965 ms (n=73). The broker retries once on a stale connection and the client timeout is 660 s — during a proxy wobble every caller pays 16–26 s before failing, and dozens of callers × retries amplify the herd (the stage-1 addendum found the same chain).
- Keepalive/idle today (Python broker, mirrored by
fleet proxy servebehavior to confirm): KEEPALIVE_S=180 health ping, broker exits after 3,600 s idle, pool per port. - Broker log has no timestamps and broker ownership has flapped between the Python broker (pool 12) and
fleet proxy serve(pool 50) with no record of which was live in an incident window (stage-1 addendum, Incident record 2026-11-04: afternoon responsiveness loss — sandbox wedged after post-boot herd (daemon restart, 88-min sampler silence); laptop path exonerated 21:00–21:28 UTC #5701).
Proposal
- Fail fast on dead target: cap total per-call failure latency at ≤2 s when the tunnel target is unreachable (bounded CONNECT attempt + one stream-level retry, no second full CONNECT inside one call). Healthy-path latency unchanged; the 660 s cap stays ONLY for the daemon-side exec duration, not for transport setup.
- Single-flight tunnel recovery: when the pool/tunnel is down, exactly one reconnect attempt in flight; concurrent callers wait on it or fail fast — never N parallel CONNECTs.
- Timestamped, identity-stamped broker log: every broker start writes one line {ts, implementation, version, pool_size, pid}; every ownership handoff is therefore reconstructable after an incident.
- Deterministic single owner: one implementation owns broker.sock (fleet proxy serve); the Python broker starts only when the fleet binary is absent (the init-ensure gate already expresses this intent — enforce it in the clients' auto-start paths too, which still spawn the Python broker directly: laptop-agent.py
_broker_roundtrip, repo-queuespawn_broker). - Keepalive policy: keep 180 s application keepalive per tunnel while pooled HTTP remains; re-evaluate under the multiplexed design (PING on the single tunnel). Idle-exit stays 3,600 s.
Acceptance
- Kill the runtime proxy mid-test: p50 failed-trip latency in trips.jsonl ≤2 s (from 16.3 s), and recovery shows exactly 1 CONNECT stampede in the watcher log.
- broker.log first line after any start is timestamped with implementation + pool size.
- No client auto-start path can spawn the Python broker while a fleet binary exists.
Activity
developing-today commentedon Nov 5, 2026
CODE-1445
dezren39 commentedon Nov 5, 2026
🤖 Agent note · handed off
dezren39 commentedon Nov 5, 2026
🤖 Agent note · handed off
developing-today commentedon Nov 5, 2026
🤖 Agent note · handed off
105 remaining items
fleet(proxy): recovery-only single-flight — healthy opens stay concur…
fleet(proxy): model-based property test for RecoveryGate + per-genera…
fleet(proxy): split connect deadline across fallback addresses; smoke…
fleet(proxy): bound DNS resolution by the transport deadline (#5758, PR
Merge pull request #6490 from developing-today/queue/10968-5758-fleet…
dezren39 commentedon Nov 6, 2026
Done: task 10968 completed by lane code-i. PR #6490 merged (merge commit c4db78f) implementing the #5758 slice: 2s transport deadline for fleet-proxy tunnel setup (DNS + connect + CONNECT handshake, incl. per-address budget split), recovery-only single-flight gate with per-generation verdicts and a model-based property test, timestamped identity-stamped broker startup line (pool default 50 unchanged), and pinned sibling-fleet broker selection in fleet-queue. Codex review rounds 1–5: every finding fixed in-PR, replied, and resolved; final round clean. Laptop: cargo test -p fleet-proxy 56 passed, fleet-queue client tests passed.