Design: custom + live audio through the façade — custom formats and realtime bidirectional (duplex) audio #1680

Parent: #1146

This is the most exploratory of the four façade-expansion designs (search, speech-to-text, image/video generation, and this one). Expect the output to be mostly inventory and options, not a settled shape.

Problem

The other façade surfaces are request/response: a caller sends a bounded payload and gets a bounded result. Audio breaks that in two ways. First, custom formats: endpoints that receive audio must cope with whatever format the sender has — codecs, sample rates, container quirks — not just the handful a chat-shaped API assumes. Second, live audio: realtime voice interfaces are bidirectional streams — you transmit while the connection stays open and it transmits back, interleaved, with interruption and turn-taking semantics. Neither fits the #1146 chat façades' translation shape directly, and it is genuinely unclear yet how much of the façade rule (ONE ingestion code path, providers converting into the supported formats passing through as much as possible, provider handling key registration and renewal) can survive contact with duplex streaming. That is what this design exists to find out.

Drew's ask (paraphrased)

Custom + live audio: receiving audio from endpoints that handle custom audio formats; sending and receiving audio through the façade; live bidirectional interfaces — you transmit while the connection stays open and it transmits back (realtime duplex).

Expanded feature design (to be designed, not yet scoped for build)

1. Inventory of live/realtime interface shapes. Catalogue what providers actually offer today for realtime audio: websocket-based realtime APIs (e.g. OpenAI-Realtime-style event protocols), WebRTC-based offerings, SIP/telephony bridges, and provider-proprietary streaming sockets. For each: transport, event/message grammar, audio encoding expectations, how sessions are created and authenticated, and how they end. Also catalogue the non-live custom-format side: which audio formats/codecs endpoints in the fleet's orbit emit or demand that a naive receiver would reject.
2. Custom-format reception. Design the façade's answer to "audio arrives in a format we didn't choose": accept-and-transcode server-side vs accept-and-pass-through to a provider that tolerates it vs reject with a precise envelope. Where transcoding would live (in-process, per the native direction of Embeddings: bake the local backend into the fork (native in-process), retire the :18752 sidecar #1554, vs a media service), format detection, and the bounded-by-default limits (design(core): bounded by default — one deadline-and-cap contract for every outbound HTTP call #1127) on stream size/duration.
3. Transport for the façade itself. Candidates: websocket upgrade on the OpenCode server (the natural fit for duplex), WebRTC (better media properties, much heavier server story), or a relay model where the façade brokers setup and media flows elsewhere. The design must weigh fork-maintenance surface (per Design: generic OpenAI façades on the server — /v1/chat/completions + /v1/responses, one translation layer, proxy doors retire #1146, façade code is conflict surface on upstream merges — keep it isolated) against latency and fidelity.
4. Normalization boundary. The hard question: how much of a provider's realtime event grammar (partial transcripts, tool calls mid-stream, interruption/barge-in signals, playback markers) can the façade normalize before it stops "passing through as much as possible" and starts silently changing behavior callers depend on — the same provider-quirk-parity concern Design: generic OpenAI façades on the server — /v1/chat/completions + /v1/responses, one translation layer, proxy doors retire #1146 raises for chat, amplified. Candidate answers range from full event translation to a thin authenticated tunnel with per-provider passthrough; the design should pick per capability, not globally.
5. Session lifecycle. Connection setup/teardown, idle and max-duration limits, reconnect/resume semantics (does a dropped duplex session resume or restart?), recording/retention posture for live audio, and how a live session relates to an OpenCode session (attached part? side channel? its own session type?).
6. Client-side requirements. What OpenCode (TUI: mic capture, playback, barge-in UX in a terminal) and OpenChamber (browser/app media stack, permissions, echo handling) would each need to be a realtime endpoint. Terminal audio is the least-charted part — the design should say honestly what a TUI can and cannot do here.
7. Key handling. Per the façade rule, the provider/façade layer handles key registration and renewal — but several realtime providers use ephemeral session tokens minted server-side; the design covers how the façade mints and hands those out without leaking long-lived credentials to clients.
8. Accounting parity. Live sessions are long and metered by time/tokens upstream; the design covers how usage lands in the route_attempt ledger discipline when a "call" is a 20-minute stream.

Open questions

• Which provider realtime shape, if any, is becoming the de-facto standard the façade should center on — or is the field still divergent enough that only a tunnel design is honest?
• Is WebRTC in scope at all for a server whose current surfaces are HTTP/SSE/websocket, or explicitly deferred?
• Barge-in/interruption: whose responsibility — provider, façade, or client — and can it survive translation?
• Does live audio belong in a session product at all (voice conversation with the agent) or is the real use case bridging external audio endpoints into tools? The inventory should surface actual demand before the design picks.
• What does "custom format" concretely mean in the cases Drew has hit — telephony codecs (μ-law etc.), raw PCM variants, container oddities? (Needs concrete examples collected during the design.)
• Latency budget: what round-trip makes duplex conversation feel live, and which hops (client → façade → provider) can the design afford?

Acceptance shape

A design doc in the repo (doc/) covering: the provider realtime-shape inventory, the transport decision (or the explicit deferral of it, with the evidence that would settle it), the normalization boundary per capability, session lifecycle, client-side requirements for OpenCode and OpenChamber, and a phased recommendation — including if the honest conclusion is "tunnel-only for now" or "not yet". Reviewed and linked from #1146.

Expansion path

From the accepted design, file implementation tickets per step — expected to be more phased than the sibling designs: (1) inventory/grounding tickets if provider shapes need live probing, (2) custom-format reception (non-live) as a separable first phase if the design supports it, (3) façade transport + session lifecycle skeleton, (4) one ticket per provider realtime backend, (5) OpenChamber client support, (6) OpenCode client support (possibly its own design spike if terminal audio proves to be the blocker), (7) accounting/ledger integration.

Acceptance criteria

• The design covers custom audio formats and realtime bidirectional audio through the façade.
• Session lifecycle, backpressure, and error handling for live streams are specified.
• An implementation follow-up issue is filed if the design is adopted.