Repository navigation
[Codex App][Remote SSH] Thread hydration blocks the app-server queue and causes an endless reconnect loop #36189
Description
What version of the Codex App are you using (From "About Codex" dialog)?
26.721.11231.0; remote Codex CLI/app-server 0.146.0.
What subscription do you have?
ChatGPT subscription (exact tier not confirmed).
What platform is your computer?
Microsoft Windows NT 10.0.26200.0 x64; remote host is Linux 6.8.0-136-generic x86_64.
What issue are you seeing?
Codex Desktop Remote SSH can enter an endless reconnect loop when it automatically hydrates several existing threads. One slow thread/resume request occupies an app-server request slot for 50-65 seconds. Additional hydration and background requests fill all six in-flight slots, queued requests expire after 30 seconds, and the whole WebSocket eventually closes with code 1006.
This affects the entire remote host connection, not only the problematic thread. New threads cannot be created or sent while the queue is blocked. Normal SSH sessions remain stable, the managed remote app-server PID remains alive, and server CPU/memory are not saturated.
The affected session files were not exceptionally large by storage standards (approximately 2-20 MB), but some JSONL records were monolithic: individual records were approximately 1.2-1.64 MB. A 2.28 MB session with only 111 records contained one 1.26 MB record. This suggests that record shape and concurrent hydration, rather than only total session size, trigger the failure.
Representative redacted Desktop log sequence:
app_server_client_request_queue_rejected inFlightRequestCount=6
method=plugin/list queueWaitMs=30003 reason=queue-expired
Request failed method=thread/resume durationMs=65397
error="Codex app-server is not available"
app_server_connection.closed code=1006 transport=websocket
proxy_command_failed signal=SIGTERM operation=app_server_proxy
SIGTERM is logged after the WebSocket closes and appears to be cleanup, not the initial cause.
What steps can reproduce the bug?
- Configure Codex Desktop on Windows with a Remote SSH Linux project.
- Create several threads in that project, including threads containing large tool outputs or other large individual JSONL records.
- Leave Remote SSH auto-connect enabled and keep those threads active/unarchived.
- Restart Codex Desktop or reconnect the Remote SSH host so Desktop hydrates multiple threads.
- Observe several concurrent
thread/resumecalls. One or more resume calls take tens of seconds. - Observe
inFlightRequestCount=6, request queue expiration after 30 seconds, WebSocket close code1006, and automatic reconnect. - Try to create or send a new thread while this is happening. The new request is blocked behind hydration requests or fails when the app-server connection closes.
The same SSH route becomes responsive immediately after the recovery steps below, without changing the network.
What is the expected behavior?
- A slow or malformed thread should only fail or delay that thread's hydration.
- Interactive operations such as
thread/startandturn/startshould not be blocked behind bulk hydration. - Thread history should be streamed, paginated, or processed with bounded record sizes.
- Remote reconnect should be bounded and should not repeatedly replay the same failing hydration workload.
- The UI should identify the offending thread and offer a reversible recovery action.
Additional information
The effective recovery was:
- Identify the thread whose
thread/resumeis blocking the queue. - Archive the offending large/abnormal thread or threads (reversible; do not delete them).
- Cancel their pending resume operations and clear queued hydration requests.
- Stop automatic reconnect instead of immediately replaying the same workload.
- Restart the managed remote daemon once to clear already in-flight server requests, then reconnect only after the offending threads are archived.
Recommended product behavior: after a hydration timeout or repeated queue saturation, show an Archive task and stop reconnecting recovery action. It should archive the identified thread, cancel pending hydration, clear the affected queue, and pause automatic reconnect until the user explicitly retries. Interactive requests should also receive priority over background hydration.
After applying the recovery above, request latency on the unchanged connection returned to approximately 1-350 ms, turn/start completed in approximately 122 ms, and the connection remained stable. Before recovery, even smaller threads took 25-31 seconds because they were queued behind the blocking resume.
Related but not duplicate:
- Codex App Remote SSH is slow and fragile behind NLB compared with VS Code Remote SSH #20128 tracks Remote SSH fragility behind a network gateway/NLB.
- Codex Desktop Remote SSH reconnect loop leaks thousands of orphaned app-server processes when hosts share an NFS home #35217 tracks leaked app-server processes during reconnects with a shared NFS home.
This report is specifically about thread hydration/backpressure causing host-wide request starvation and a reconnect loop.
Activity
github-actions commentedon Jul 30, 2026
Potential duplicates detected. Please review them and close your issue if it is a duplicate.
Powered by Codex Action
yufeinever commentedon Jul 30, 2026
I reviewed #35096. There is clear overlap: both are triggered by Windows Codex Desktop automatically restoring Remote SSH threads.
However, #36189 captures a narrower backend failure mode that #35096 does not document:
thread/resumerequests saturate all six app-server request slots, queued requests expire after 30 seconds, the WebSocket closes with code1006, and automatic reconnect repeatedly replays the same workload. The UI does not necessarily become fully unresponsive, but new threads cannot be sent.#36189 also includes request timings, evidence of monolithic 1.2-1.64 MB JSONL records, and a reproducible recovery: archive the offending threads, cancel hydration, stop reconnecting, clear the queue, and restart the managed daemon once.
These issues may share the same underlying thread-restoration regression, but I am leaving #36189 open as the app-server/backpressure and reconnect-loop case unless maintainers confirm they should be consolidated.
tpn commentedon Aug 2, 2026
Confirming this on a Linux server & Windows ChatGPT App setup, Windows
connecting to the Linux box via "Connections" & ssh, with server-side evidence
from three related incidents. Remote host: Ubuntu (kernel 6.17), managed
standalone app-server 0.145.0 and later 0.146.0; client: Codex/ChatGPT Desktop
on a separate Windows workstation over SSH; current version 26.727.51351 but
this occurred recently on earlier versions over the past week.
Symptom match. Desktop entered the endless reconnect loop described here:
SSH itself fine, but Codex never finished connecting. At the time of the most
recent incident the remote session store was 16 GB across 6,621 rollout files —
largest 492 MB, with 61 files ≥ 10 MB. These are far past the 2–20 MB files
reported above, which may explain why our failure mode progressed further
(below).
Server-side observations across the three incidents (Jul 25, Jul 28, Aug 2):
codex app-server proxyon the remotehost via SSH. When the app-server stops responding, the proxies never time out
and never exit. We accumulated 4,076 orphaned proxy processes in one
incident and 3,506 in another (spawn cadence ~every 10–50 s), with the
app-server's unix-socket accept queue saturated at 2,731 and 2,906 pending
connections respectively. Same leak shape as Codex Desktop Remote SSH reconnect loop leaks thousands of orphaned app-server processes when hosts share an NFS home #35217, no NFS involved.
~7 cores (~46 k futex syscalls/s, SIGTERM ignored, required SIGKILL — matches
app-server: degraded broker catches SIGTERM then deadlocks on a futex (needs SIGKILL); healthy broker exits <1s #34142), once fully deadlocked at 0% CPU while holding the listener. RSS
reached 7.6 GB with another 3.8 GB in swap before the hangs.
restarted it (matches Persistent app-server and pid-update-loop disappear without restart, leaving stale daemon state on Linux #35295); the next incoming proxy then auto-spawned an
unmanaged replacement (ppid 1) that the daemon refuses to manage, so recovery
required manually killing it.
On the six-slot limit. I searched codex-rs (0.146.0) for the limiter: there
is no 6-permit semaphore anywhere in the app-server — request pools are 256/1024
and the per-connection
ConnectionRpcGateis unbounded — and the stringinFlightRequestCountdoesn't appear in this repo. So the six in-flight slotsand the 30 s queue expiry appear to live in the closed-source Desktop client,
meaning affected users have no configuration or self-build escape hatch on
either end.
Mitigation data point. Archiving old/large rollouts (active set reduced from
16 GB / 6,621 files to ~4.7 GB, largest active file now ~10 MB) stopped the
reconnect loop immediately. That's consistent with the hydration-cost theory:
six slots are fine when
thread/resumeis fast; they're fatal when a handful ofresumes take 50+ s.
Requests:
configurable — or better, deprioritize bulk thread hydration behind
interactive requests so a slow resume can't starve new-thread creation.
thread/resume(0.146.0 already added paginatedhistory for forks; resume seems like the natural next candidate).
codex app-server proxyfail fast (bounded connect/handshake timeout,then exit non-zero) when the app-server is unresponsive, so a wedged server
degrades into fast failures instead of thousands of orphaned processes and a
saturated accept queue.
Happy to provide the futex traces, accept-queue captures, or process trees from
any of the three incidents if useful.
raka08 commentedon Aug 23, 2026
Confirming the same user-visible failure on a macOS controller → Windows Remote SSH host setup.
On the same host and SSH configuration:
Loading task....Failed to resume chat — Codex app-server is not available.Connection to [host] was lost. Reconnecting..., and retrying does not open that long chat.This long-vs-short control makes a general SSH authentication/connectivity problem unlikely and points to the thread history hydration/resume path described in this issue. The expected behavior is for a large thread to load incrementally, or fail only that thread with a specific recoverable error, without dropping and replaying the entire remote connection.
I have screenshots of both the loading state and the exact errors. I am intentionally not posting thread IDs, host names, local paths, raw logs, or chat data publicly; sanitized diagnostics can be supplied privately to OpenAI maintainers if requested.
northern-penguins commentedon Oct 10, 2026
Additional macOS → Linux Remote SSH evidence on app-server 0.162.1, with an important distinction between the initial delay and the Desktop queue amplification. This is supplementary evidence, not a confirmed reproduction of the exact hydration or large-catalog root cause in this issue.
Environment: official Desktop 26.1007.21159 / build 20052, macOS arm64; two Linux x86_64 standalone app-servers 0.162.1. The hosts share a durable Codex home but have separate instance-local SQLite/runtime/socket directories. SSH is carried over a platform WebSocket tunnel. No application patch was deployed, and no running task was interrupted for these checks.
Initial delay: server response already enqueued, Desktop routes it about 31 seconds later
We correlated the same RPC IDs in the remote app-server's SQLite logs and the Desktop main-process log. During a six-minute incident window, 127 matched responses belonged to the affected server's connection 0; 121 arrived at Desktop more than 20 seconds after server receipt. These are matched samples, not a count of every RPC. Small
account/read,config/read, and filesystem/process RPCs were affected, not only history reads.Three
process/spawnrequests are particularly useful. They read existing plugin PNG assets of 7,216 / 12,063 / 10,819 bytes with/bin/cat:process/exitedevent log (UTC)Date: 2026-10-10. In 0.162.1 process_exec_processor.rs,
send_response(..., ProcessSpawnResponse {})is awaited before process output/exit collection. Therefore these response messages were already enqueued by the exit-event timestamps. This excludes the actual small-file command execution as the approximately 31-second delay.However, OutgoingMessageSender hands responses to a channel with
write_complete_tx: None. Enqueue completion is not proof of socket flush. Desktopresponse_routedis also not the raw first-byte arrival timestamp. We cannot retrospectively separate:Amplification is a separate, confirmed layer
The Desktop later logged
App server request queue is full, with 8 in-flight requests and an interactive queue limit of 64 in this build. Athread/queue/listspent 146,591 ms queued, followed by 33,815 ms after dispatch. The initial delay was already present on other requests withqueueWaitMs=0. Increasing timeouts or changing request priority alone would not explain or eliminate that initial delay.A
fs/createDirectoryresponse appeared asresponse_orphanedafter the UI had timed out. A later Desktop reconnect also encountered1006and a proxyBroken pipe; these are later observations, not proof of the first delay's cause.Read-only controls, after the incident
At 11:06 UTC, we opened a new independent proxy connection to each existing app-server, without starting/resuming/subscribing to a task. We used only initialize,
fs/getMetadata,configRequirements/read, andaccount/readwithrefreshToken=false:All 12 SSH control RPCs succeeded. These are post-incident controls on new connections, not a reproduction on the old Desktop stream, and do not rule out historical network or client backpressure. No task data, history response, full plugin catalog, or credentials were retained by the probe.
We checked #41573 and #43525 as well. The 0.162.1 WebSocket writer still serializes and sends each outgoing message before selecting the next, so head-of-line blocking remains a plausible mechanism. But the available affected-server logs for the incident contain no
plugin/list,app/list, orapp/list/updated, and the sampled icons are small. A large catalog cache on disk is not evidence that it was transmitted. We therefore do not claim the plugin catalog caused this incident, or that the history file size alone proves hydration was the initial blocker.Questions for maintainers
responseReceiveDurationMs, large inbound byte counts, and notification delivery lag. Can those metrics be exported through a supported diagnostics interface so we can distinguish return-stream backpressure from Desktop dispatch delay?We have deliberately not archived history, disabled plugins/apps, removed sockets/databases, retried effectful RPCs, or restarted active servers. Only sanitized timings, byte counts, and public source references are included; no usernames, host endpoints, thread IDs, prompts, raw logs, or account identifiers are disclosed.