Repository navigation
App-server still drops remote-control clients when outbound queue fills (128 messages) — #18203 fix missed the code path #37526
Description
Summary
codex app-server still disconnects remote-control clients mid-turn with disconnecting slow connection after outbound queue filled on codex-cli 0.147.0 (and the same code exists on current main and rust-v0.148.0-alpha.5). This is the same symptom as #18203, which was closed as fixed by #19246 — but that PR only raised the WebSocket writer buffer. The per-connection outbound message channel that actually triggers the disconnect is still 128 messages, so the bug is unfixed.
Environment
- codex-cli 0.147.0 (standalone package, Linux x86_64)
codex app-serverwith remote control enabled; phone + desktop apps connected through the chatgpt.com relay (wss://chatgpt.com/backend-api/wham/remote/control/server)- model routed through a local OpenAI-compatible proxy (
opencode-go/deepseek-v4-flash,reasoning_effort=max)
Root cause
CHANNEL_CAPACITY is still 128 (verified identical on main, rust-v0.147.0, and rust-v0.148.0-alpha.5 as of 2026-08-07):
// codex-rs/app-server-transport/src/transport/mod.rs (line 25)
pub const CHANNEL_CAPACITY: usize = 128;Disconnect path unchanged (codex-rs/app-server/src/transport.rs):
if connection_state.can_disconnect() {
match writer.try_send(queued_message) {
Ok(()) => false,
Err(mpsc::error::TrySendError::Full(_)) => {
warn!("disconnecting slow connection after outbound queue filled: {connection_id:?}");
disconnect_connection(connections, connection_id)
}
...Reproduction
Run two threads concurrently so both stream reasoning deltas to one remote-control connection (each max-effort thread can emit hundreds of events/sec; relay latency lets the 128-slot queue fill). The app-server then kills the client:
20:23:28 WARN transport | disconnecting slow connection after outbound queue filled: ConnectionId(25)
20:23:28 WARN transport | dropping message for disconnected connection: ConnectionId(25)
20:23:28 INFO client_tracker | forwarding remote control connection closed transport event connection_id=ConnectionId(25)
Both the phone and desktop apps show "connection lost"; the thread keeps running server-side but the client session is dropped.
Expected behavior
A slow-but-alive client should not be disconnected because of a burst. The outbound queue should be sized for streaming workloads, or the writer should apply backpressure (await send) and only disconnect after a timeout.
Suggested fix
- Raise
CHANNEL_CAPACITYfor remote-control connections (the WebSocket writer buffer was raised to 64K in Increase app-server WebSocket outbound buffer #19246), or - Replace
try_send+ immediate disconnect with async backpressure plus a disconnect timeout.
References
- App-server disconnects remote TUI mid-turn when outbound queue fills (128 messages) #18203 (original report, closed 2026-04-24)
- Increase app-server WebSocket outbound buffer #19246 (merged "fix" that missed this code path)
- Evidence comment on App-server disconnects remote TUI mid-turn when outbound queue fills (128 messages) #18203: App-server disconnects remote TUI mid-turn when outbound queue fills (128 messages) #18203 (comment)
Activity
niklassheth commentedon Aug 9, 2026
I'm experiencing this issue, it's incredibly frustrating because every time the connection drops my side chats expire.
jdcodes1 commentedon Aug 18, 2026
Verified on
main(1f41cc5): both halves of the analysis hold.CHANNEL_CAPACITYis still 128 (app-server-transport/src/transport/mod.rs#L25, re-exported into the app-server), and the disconnect path is byte-identical —try_send→TrySendError::Full→ immediatedisconnect_connection(https://github.com/openai/codex/blob/1f41cc5d92/codex-rs/app-server/src/transport.rs#L156-L163). So #19246 enlarged the layer below the one that kills the connection; the mpsc hop above it still hard-drops at 128, and two max-effort reasoning streams through a relay can outrun that trivially.Beyond raising the constant, the structural issue is that disconnect is the only overflow policy. For a remote-control observer, the queue is full of deltas whose value decays instantly — the correct degradation is coalescing/shedding, not connection teardown: drop oldest delta messages (never lifecycle/completion events) when full, or collapse consecutive deltas for the same item into the latest one. That bounds memory identically while making slow relays lossy-but-alive — a phone missing some reasoning frames is fine; a phone being disconnected mid-turn (then re-syncing everything) is both worse UX and more total bandwidth. A capacity bump alone just moves the cliff; the reproduction with two concurrent threads will find any fixed constant.
charles1w commentedon Sep 7, 2026
Windows confirmation (desktop 26.901.4073, Sept 5–7): captured 17
disconnecting slow connection after outbound queue filledevents, each followed by the same remote client's closure 0.1–1.4 ms later. RustDesk remained responsive during observed Codex Remote drops. In 16/17 incidents there was no host-relay reconnect within ±35 seconds; independently, both IPv4/IPv6 probes passed around 16/17.The confirmed immediate failure is per-client queue overflow triggering connection teardown. Likely trigger: streaming bursts outrunning relay/client consumption; writer or acknowledgement stalls remain possible but unmeasured. Queue occupancy, drain rate and ACK latency would distinguish these. This supports fixing backpressure/overflow handling rather than resetting host networking.
These captures predate 26.901.6511; that build has not been revalidated.
charles1w commentedon Sep 10, 2026
Follow-up to my earlier Windows queue-overflow confirmation: the persistent inability to reconnect had an additional, separately reproduced cause: slow
thread/listscanning. This does not establish what originally filled the outbound queue.On app-server
0.153.4, Android's list request omitsuseStateDbOnly, causing the host to parse huge subagent histories before excluding them. Controlled listing exceeded 25 seconds; early exclusion returned the same 50 tasks in 1.15 seconds (indexed: 27 ms).A live workaround preserves old histories via retained NTFS files and read-through links that the scanner skips. Task loading now takes 1.21 seconds and the phone connects; no app restart or manual network reset. The code-level remedy is indexed Remote bootstrap and source filtering before preview parsing.
Sanitized reproduction and workaround details. The previously captured queue-full disconnects remain valid and require their own fix.
arkycop commentedon Oct 3, 2026
Fresh Windows-to-macOS confirmation on 0.159.0-alpha.12.1: command-output bursts coincide with queue-full disconnects
Adding sanitized evidence from October 3, 2026 (UTC). This reproduces the queue-overflow disconnect mechanism described in this issue on a newer bundled app-server and a Windows Codex App host with macOS Codex remote projects.
Environment and impact
codex-cli 0.159.0-alpha.12.1.26.930.2377.0; Microsoft Windows NT 10.0.26300.0, x64.26.930.21537; macOS27.2, Darwin27.2.0, ARM64.Correlated failures
All times are October 3, 2026, UTC.
item/commandExecution/outputDeltaitem/commandExecution/outputDeltaRepresentative Windows native messages:
The macOS desktop log then records close code
4000with reason:The installed macOS app sends a logical stream heartbeat every 30 seconds. An unknown-stream pong causes this close and a scheduled reconnect, explaining why the visible Mac drop can lag the Windows queue-full event.
Measurement and source boundary
Windows native SQLite logs were read in read-only mode with short-interval polling. Outgoing notification records were classified by protocol method and grouped by native second/nanosecond timestamps. These are app-server notification counts, not console-line or byte counts. Counts cover the server's logged outgoing notifications; they do not independently prove every counted notification targeted the disconnected client. The transport warning separately establishes that the client's outbound queue filled.
The official source tag matching the installed bundled CLI still has:
This establishes the disconnect mechanism and strongly implicates command-output bursts. Live relay acknowledgments and sustained packet timing were not captured, so acknowledgment latency, network latency, or another drain-rate defect may also contribute. Matching the published version tag is not a byte-for-byte binary verification. Other historical sequence-gap and pong-timeout failures are not assigned this cause.
Checks and mitigation
Expected behavior and request
A burst of command output should not repeatedly tear down an otherwise live remote project connection. Please investigate bounded backpressure/overflow handling that preserves required response and lifecycle events, along with queue occupancy, drain-rate, and relay-ACK diagnostics. The practical output-capture workaround is useful, but users should not have to control normal command verbosity to keep the desktop remote connection alive.
This report intentionally excludes usernames, hostnames, internal addresses, account/enrollment identifiers, project names, conversation content, credentials, and full raw logs.
Follow-up: October 3 morning recurrence
The workflow mitigation has not eliminated the client symptom. Account-wide output-capture guidance remained installed on both macOS and Windows, and the app/CLI versions were unchanged. The Windows desktop app-server remained alive. Mac logs recorded two further code-4000 closes with reason
remote control app-server stream became unknownat 2026-10-03T15:22:46.564Z and 15:27:17.882Z. A separate authenticated-workspace-change close preceded a Mac app restart around 15:21 UTC; it is not counted as a transport failure.No matching Windows queue-full warning was recovered for these two morning events, so they are not proven to have the earlier overflow cause. A subsequent 15:28:20.449–15:28:58.514 UTC sample recorded 1,051 command-output notifications, peaking at 228 in a 100-ms bucket, without a transport warning. A further 15:34:46.850–15:37:46.945 UTC capture recorded 146 agent-message deltas, no commandExecution/outputDelta notifications, and no transport warning or closure; the Mac also recorded no additional remote closes through 15:43:11 UTC.
The earlier timestamp-correlated queue-overflow evidence remains valid. These later observations establish that output-capture guidance is only a partial workaround, while leaving the specific cause of the morning reconnects unclassified. A deterministic burst/slow-ACK reproduction and transport diagnostics are still needed.