Repository navigation
Warm-session proxies: hold CLI/language-server sessions open instead of spawning per request #480
Description
Related: #479 adds bin/muse-proxy.mjs, the newest spawn-per-request proxy — this issue is its (and codex-proxy's) v2 design question.
Problem: every proxy request pays a full cold start
The CLI-backed proxies in bin/ (codex-proxy, and now muse-proxy) spawn a fresh CLI process per chat-completions request and flatten the entire message history into a single prompt. Measured on muse-proxy (2026-10-21):
- ~5–8 seconds of latency floor per request before model time — process spawn + runtime boot + agent/session setup dominates; generation is the tail.
- ~27k input tokens of CLI session context (system prompt, skills) are built and billed on every call, trivial prompts included.
- No context or prompt-cache continuity: request N+1 in a conversation re-sends and re-pays everything request N established.
Verified while filing: bin/codex-proxy.mjs contains no session handling at all — no session, resume, or conversation logic anywhere in the file; each request is an independent codex exec. muse-proxy is the same shape by design (it mirrors codex-proxy).
Existing proof of concept in this repo
bin/antigravity-ls-proxy.mjs does not spawn per request: it attaches to a long-running language-server process (managed LS ports) and multiplexes requests over that warm process. That shape works in production here today. The CLI proxies are the odd ones out.
Avenues
(a) Muse — hold a warm muse serve host
The Muse CLI ships muse serve ("Serve an MSP session host over stdio") and muse schema (exports the MSP wire schema, JSON Schema or TypeScript). A v2 muse-proxy could:
- Spawn
muse serveonce and keep it warm for the proxy's lifetime. - Open real sessions on the host per conversation, so session state, context, and any server-side caching persist across requests in that conversation.
- Translate OpenAI chat-completions ⇄ MSP using the exported schema rather than scraping
exec --jsonoutput.
Unknowns to resolve in the spike: MSP session lifecycle (create/resume/close), concurrency model (one in-flight turn per session?), and whether prompt caching actually engages across turns on the plan allowance.
(b) Codex — key resumable exec sessions by conversation ID
The Muse's exec sessions are resumable. codex-proxy could:
- Accept/derive a conversation key (client-supplied session/conversation id, falling back to a hash of the system prompt + first message) and map it to a CLI session id.
- Resume that session for follow-up requests, sending only the new turn instead of the flattened full transcript, so prompt cache and context carry over.
Unknowns: exact resume semantics of the installed CLI version, whether resumed sessions preserve cached prefixes measurably (check usage output across two turns), and session id storage/cleanup.
Trade-offs to design around
- Statefulness: the proxy becomes a session owner. Restarts drop warm sessions; clients must tolerate a cold fallback (re-flatten and start fresh) as a normal path, not an error.
- Serialization: a warm session likely handles one in-flight turn at a time → per-conversation request queues, plus a small pool of warm sessions for concurrent conversations, with sane caps.
- Hygiene: a session's cwd and accumulated context bleed across requests if keying is sloppy. Conversation keys must be strict (never share a warm session across different clients/projects by accident), and idle sessions need a TTL + eviction.
- Observability: keep reporting real per-run usage (the CLIs already emit it) and add warm/cold markers so the win is measurable, not assumed.
Spike acceptance criteria
- Warm per-request latency measured against the ~8s cold baseline on the same prompt set (report both, same machine, same models).
- Cache/context continuity demonstrated across two requests in one conversation: the second request answers a follow-up that requires the first turn's context without the client re-sending history, and/or usage numbers show the reused/cached input share.
- Two concurrent conversations stay isolated (no cross-talk of context) and a third waits or cold-starts per the pool policy.
- Killing the warm host/session mid-conversation degrades to a successful cold request, not a client-visible error.
- No regression in the OpenAI-compatible surface (
/v1/models, streaming + non-streaming chat completions) for either proxy the spike touches.
Activity
feat(agy-proxy): warm conversation reuse keyed by system+first user+m…
warm: keep-alive backend/upstream pools for design-code + clef proxies (
codex-proxy: warm app-server daemon transport as lead route (#480)
dezren39 commentedon Oct 24, 2026
Warm-session program — results (2026-10-24, Drew's directive)
All timings use one named model per provider family, held constant across before/after, same two-turn script. Canonical laptop baselines (gpt-5.6-luna): native openai provider turn 1 = 7.5s / turn 2 = 3.0s; spawn-per-request codex-proxy turn 1 = 61.6s / turn 2 = 49.6s.
codex-proxy — TWO warm mechanisms shipped
codex exec resume(thread_id fromthread.started, prefix-validated, sessions.json LRU 64/2h, full fallback). Turn 2 69.8s → 39.5s (gpt-5.6-luna-low); ~93% cache-hit input. CLI/MCP startup still dominates.~/.codex/app-server-control/app-server-control.sock), conversations mapped to daemon threads with the same keying discipline; exec-resume → exec fallback chain proven live with the daemon SIGSTOPped (request still answered). app-server proxy, gpt-5.6-luna, turn 1 = 15.80s, turn 2 = 2.69s, turn 3 = 2.69s — turn 2 at parity with native 3.0s (21,248 of 21,466 input tokens cache-hit). Remaining turn-1 cost ≈ one-time thread/start (~5.1s) + the daemon's full agent context on first inference; amortizes to zero from turn 2. Tests 18/18.Note: a concurrent fleet proxy-supervisor rollout moved the live laptop proxy to port 19744 (
serveRestartRequired: truein endpoint.json — OpenCode still aimed at 18744; supervisor/fleet business to reconcile, proxy itself live and correct).muse-proxy — warm sessions on
muse serve(PR #769)Persistent MSP session host driven over raw NDJSON JSON-RPC (stdlib only, no SDK dep): keyed conversations send only the newest turn, state persisted (warm-sessions.json, LRU 64/2h), session/resume re-attach, spawn-per-request exec fallback intact. Sandbox, muse-spark-1.3: turn 1 55.6s → 13.2s (before-figure contention-inflated; honest uncontended exec ≈ turn 2's 19.9s), turn 2 19.9s → 8.5s, turn 3 7.2s; turn-2+ input is ~99% cache reads (~230–260 new tokens/turn vs a full 26.7k re-read). Tests 9/9. Deployed + live on the sandbox (health reports warm host up); no laptop deployment exists (separate parity workstream).
design-code-proxy + clef-proxy — keep-alive pass (PR #768)
design-code: keep-alive agent pools for backend (18744/18745) + AMR upstream incl. streaming; routing untouched. clef: found and fixed per-request disk reads — 3 Cloudflare credential files re-read on every request, now an mtime-checked cache (undici fetch pooling kept deliberately: raw https.request would ignore the env egress proxy). Tests 11/11. Deploy caveat: laptop units exec a nix-store snapshot; files are in
~/code/binand the PR is open, but the running services pick the code up only after a home-manager rebuild — the usual./swas deliberately not run by the worker (dirty tree + tailscale restart). Merge + rebuild activates.antigravity-ls-proxy — verified already warm, no change
The reference pattern holds: proxy attaches to a language-server process alive 13d 8h (PID 21206, port 34809); no per-request spawn in the request path; two fresh-conversation calls at 2.125s / 2.066s. Warmth = the LS process never dies; default mode still flattens caller history into one RPC (continuity lives in cascade mode only). Sandbox has no LS deployment (gap owned by provider-parity work).
agy-proxy — built, then SHELVED by Drew's consolidation (PR #767 closed unmerged)
Warm conversation-reuse was completed, then Drew consolidated Antigravity to antigravity-ls only (antigravity-agy disabled fleet-wide, kept in config as fallback). PR #767 closed without merging, branch
warm/agy-proxy-conversation-reuseretained; the one sandbox deployment was reverted to the origin/main file and the service verified healthy; the laptop was never deployed to.CommandCode — out of this program's scope
Its ACP resident-session retrofit landed under its owning workstream (turn 2 = 1.86s) and served as this program's reference bar for what warm should look like.
Open follow-ups: merge order #766 → #773 (stacked), plus #768, #769; laptop home-manager rebuild to activate #768's code; supervisor to reconcile OpenCode's 18744 pointer with the proxy's supervised 19744 endpoint; optional pre-warmed spare codex threads to erase the ~5s thread/start on turn 1.
Merge pull request #766 from developing-today/warm/codex-proxy-sessions
Merge pull request #773 from developing-today/warm/codex-proxy-app-se…
Merge pull request #769 from developing-today/warm/muse-proxy-serve-pool
Merge pull request #768 from developing-today/warm/keepalive-design-clef
Merge pull request #875 from developing-today/warm/claude-code-proxy
13 remaining items
dezren39 commentedon Nov 1, 2026
🤖 Agent note · picked up
test(muse-proxy): concurrent-conversation isolation for #480 (stub MS…
dezren39 commentedon Nov 1, 2026
🤖 Agent note · done (verification pass, task 4810)
Verification results — #480 spike criteria (2026-11-01, laptop, main dad9ddd)
1. Warm latency vs cold baseline — ✅ verified. PR #769 measured the same two-turn script with the real muse CLI: warm turn 2 8.2s vs 19.9s for the same turn via cold
muse exec(which also re-read 26,708 fully-uncached input tokens). PR #773 measured the codex app-server warm route at 2.15s turn 2 vs 56.07s for the exec-resume turn 2 on the same script (gpt-5.6-luna, laptop). Both are far under the ~8s-per-request cold floor this issue filed against, on the warm turn that repeats.2. Cache/context continuity — ✅ verified. muse-proxy test
warm serve path: second turn sends only the new user message (#480)(passes today, 27/27 suite) asserts one MSP session, tail-only turn 2, andcached_tokens: 40from the MSPcacheReadTokens; the live figure recorded inbin/muse-proxy.mjs(line ~298) and PR #769 is turn-2 26,225 cacheReadTokens of 26,455 input tokens (~99%). codex-proxy'ssecond turn resumes the warm session with only the new tailcovers the same shape on the exec-resume route.3. Concurrent-conversation isolation + pool policy — ✅ verified, test added. Keying is strict (sha256 of model + system prompt + first user message; a changed system prompt bypasses resume — existing codex test), pools are capped (muse/codex session maps LRU 64 + 2h TTL; agy live workers cap 32 + idle eviction), and turns serialize per conversation key. That isolation had no explicit two-conversation test, so PR #3764 adds
two concurrent conversations stay isolated: separate sessions, tail-only turns (#480)— two interleaved conversations, two MSP sessions, no cross-conversation history in any turn (passes). (Its first run also exposed that the test stub returned one shared session id for every session — the stub now mints unique ids like real MSP.)4. Warm-host kill mid-conversation — ✅ verified, test added. codex already had
daemon death mid-turn falls back to exec and still answers (#480). muse only had a serve-start failure test, so PR #3764 addswarm host killed mid-turn degrades to a successful cold exec request (#480): the stub host SIGKILLs itself mid-turn on turn 2 before any delta, and the request still answers via the cold exec path (passes). agy's equivalent is structural: a dead resident worker is killed/never reused and a failed resume retries once as a fresh conversation (existing #678/#683 tests, passing today).5. No OpenAI-surface regression — ✅ verified for muse-proxy + agy-proxy; codex-proxy split out. Today on the laptop:
bin/muse-proxy.test.mjs27/27 pass,bin/agy-proxy.test.mjs13/13 pass (both include streaming + non-streaming shapes). The codex suite, however, hangs on clean main: first test alone takes 60.3s and the suite never gets past the #3114 CLI-transport test (239s,Promise resolution is still pending) — pre-existing, unrelated to this change set, and possibly the same family as issue #2157 (no turn deadline on exec transports). Split out as issue #3769 with repro evidence instead of being fixed inside this packet, per the packet instruction.agy cached-token question — SETTLED: the figure exists, and now it is emitted
The real agy CLI's stream-json
result.usagecarriescache_read_tokens(cumulative over the session; full key set observed on the laptop 2026-11-01:input_tokens, output_tokens, thinking_tokens, cache_read_tokens, total_tokens). PR #3764 makes agy-proxy delta it per turn and emitprompt_tokens_details.cached_tokensin both response shapes when > 0 — the opencode#394 contract the sibling proxies already implement — with the settled answer recorded in thebin/agy-proxy.mjsheader. New test:agy cache_read_tokens surfaces as prompt_tokens_details.cached_tokens in both shapes (#480).Merge pull request #3764 from developing-today/queue/4810-480-warm-ve…