Remote-control transport couples iOS latency to shared queues and backpressure #27592

Summary

Remote control between ChatGPT iOS and Codex Desktop is currently internet-mediated, and the app-server transport layer has at least two queue-coupling points that can turn ordinary WAN jitter into very visible mobile lag.

During investigation and local testing, I found one concrete root cause for slow iOS -> desktop steering and one remaining root cause for desktop -> iOS latency / reliability under load.

A prototype fix for the inbound path exists locally on branch codex/remote-control-reliability-fixes at commit 782f5ea585.

Related broad symptom report: #22773

Current architecture note

This path is not currently a direct LAN/local-network transport. The iOS <-> desktop control path rides the remote-control internet/websocket flow, so transport stalls are amplified by WAN variance.

That does not explain the full severity of the lag by itself. There are also transport-layer blocking points in the desktop host.

Root cause 1: inbound iOS -> desktop steering was serialized behind app-server queue waits

Before the local fix, the remote-control websocket reader held the ClientTracker lock for the whole read loop and then awaited handle_message(...), which could await transport_event_tx.send(...) for up to 30 seconds.

That meant:

• one blocked IncomingMessage forward could stop the reader from consuming subsequent websocket frames
• steering messages from iOS could sit behind a bounded app-server queue instead of being read immediately
• pong traffic was also delayed behind the same serialized path

Relevant code paths:

• codex-rs/app-server-transport/src/transport/remote_control/websocket.rs
• codex-rs/app-server-transport/src/transport/remote_control/client_tracker.rs

Observed effect

After splitting websocket frame reads from transport-event forwarding with an internal inbound inbox, steering from iOS to desktop became effectively immediate in local testing.

Root cause 2: outbound desktop -> iOS routing can still head-of-line block unrelated traffic

The remote-control reliability change switched remote-control outbound delivery from disconnect-on-full to wait-on-full. That is directionally correct for reliability, but it is currently awaited inline from the single outbound routing loop.

That means one slow remote-control writer can hold up the shared app-server outbound router instead of just slowing that one mobile client.

Relevant code paths:

• codex-rs/app-server/src/transport.rs
• codex-rs/app-server/src/lib.rs

Why this matters

This turns per-client backpressure into global latency. Under load, a slow iOS remote-control stream can delay notifications/responses for unrelated connections.

Remaining design risk: message-slot accounting instead of byte accounting

Remote-control queue/buffer capacity is now much larger, which helps avoid hard disconnects, but capacity is still counted in message slots rather than bytes.

That means a burst of large envelopes can still:

• consume a lot of memory
• create bursty latency instead of clean flow control
• hide overload until the process is already buffering too much work

Relevant code paths:

• codex-rs/app-server-transport/src/transport/remote_control/mod.rs
• codex-rs/app-server-transport/src/transport/remote_control/websocket.rs

Suggested follow-up

1. Keep the inbound reader/forwarder split so websocket reads are never blocked on transport_event_tx backpressure.
2. Change remote-control outbound routing so a full remote-control writer does not block the shared outbound router for unrelated clients.
3. Consider byte-based accounting (or at least hybrid byte + message caps) for remote-control outbound buffering.
4. Add field metrics/logging for:
   • transport-event queue depth
   • remote-control writer queue depth
   • unacked envelope count
   • ACK latency
   • disconnect / timeout reason
5. Decide explicitly whether the product should eventually support a LAN-short-circuit transport for local iOS <-> desktop control, since the current architecture always pays internet-path latency.

Validation

The inbound prototype fix passed focused tests in codex-app-server-transport, and local manual testing showed a clear improvement in iOS -> desktop steering latency.