Repository navigation
Severe WS stalls in 0.153.4: live Ping/Pong still trips the idle watchdog (Windows loopback-only reproduction) #43022
Description
Urgent reliability report
Please urgently investigate the distinction between WebSocket transport liveness and application-event silence in the Codex CLI idle watchdog.
I have now reproduced the relevant behavior using the actual installed Windows codex-cli 0.153.4 executable, an isolated CODEX_HOME, and a synthetic server listening only on localhost. There are no images, no real model inference, no account credentials supplied, and no production proxy in the model-request path.
The peer sends protocol Ping frames, and Codex demonstrably replies with Pong. Nevertheless, Codex reports idle timeout waiting for websocket and abandons the WebSocket. Sending application-level JSON heartbeats instead, or increasing only the configured idle threshold, allows the same delayed response to complete over WebSocket.
I am seriously dissatisfied with the repeated multi-minute stalls in real work. Codex CLI becomes practically unusable for the affected task when a short local command completes and the next model step repeatedly disappears into timeout/retry cycles. A paid coding workflow should not require users to reverse-engineer transport internals or build a mock server to understand why “working” produces no useful progress. Please provide urgent triage, an actionable supported mitigation, and a prioritized correction to the liveness/progress handling.
This is a separate failure mode from my image-history report #43015. It is closely related to #39771, with additional current-stable Windows evidence and a small local reproduction. Please consolidate into the existing tracker if appropriate; the purpose is to provide reproducible evidence, not duplicate noise.
Environment and production symptom
| Field | Value |
|---|---|
| Installed CLI | codex-cli 0.153.4 |
| Platform | Windows x64, OS build 10.0.26100 |
| Shell | PowerShell 7.6 |
| Production provider/auth | Built-in OpenAI provider; ChatGPT Pro authentication |
| Production model | gpt-6-astra, ultra selection |
| Production network | Local v2rayN proxy; disclosed separately from the loopback reproduction |
| Audited source | rust-v0.153.4, commit 3d2ee51ca2d5db578f328aa75e20aa22c0197c9a |
| Lab provider | Custom, unauthenticated localhost Responses provider, solely to run the installed binary against a synthetic peer |
The production session was doing primarily text/code maintenance. During the inspected resumed period, no new image input or image-viewing output was recorded. The reconstructed post-compaction history contained only one old image of 133,762 data-URL characters and approximately 1.48 MB of serialized history items overall. This is a structural history measurement, not a capture of every actual WS request.
The most recent completed input was about 278,055 tokens, with 275,840 cached, against a reported context window of 828,400. This is not evidence of a full context window or a tens-of-megabytes image request.
Recent local command executions completed successfully in approximately 0.36–0.41 seconds. The process remained alive; the inspected system snapshot showed about 10% CPU load and ample free memory. These snapshots do not rule out every environmental problem, but they do not support blaming a locally hung command or resource exhaustion for the recurring stream waits.
Observed production errors
On September 5, 2026 (UTC), the same turn recorded:
| Time | Error |
|---|---|
| 13:18:13 | websocket closed by server before response.completed |
| 13:23:55 | idle timeout waiting for websocket |
| 13:34:19 | idle timeout waiting for websocket |
| 13:45:04 | idle timeout waiting for websocket |
| 13:58:09 | idle timeout waiting for websocket |
Reconnect handshakes typically succeeded in approximately 1–2 seconds. For example, the 13:58:09 timeout was followed by a successful handshake at 13:58:11. After a tool result at 13:15:09, the next recorded model tool call was not until 13:39:30. That interval includes retries and intervening input/agent events; it is not a pure upload-time measurement.
I distinguish the first server-close error from the later client idle errors. I have not captured enough production wire evidence to prove that every one of those online failures occurred while the upstream model was still healthy. The controlled experiment below establishes the client's behavior without relying on that assumption.
Controlled local reproduction: three cases
The test runs the unmodified installed CLI against a loopback-only synthetic peer. The peer acknowledges the request with response.created, delays the final answer for 3.2 seconds, then sends valid Responses events containing OK.
For a short, bounded test, the idle threshold is 1,500 ms instead of the production default 300,000 ms. The comparison uses the same binary and response sequence. stream_max_retries=0 avoids wasting time on repeated attempts; the CLI may immediately use HTTP fallback. This is a deliberately scaled reproduction of the same wait path, not a claim that production used a 1.5-second timeout.
| Case | Heartbeat during the 3.2-second delay | Idle threshold | Peer observed | Result |
|---|---|---|---|---|
| A | WebSocket protocol Ping | 1,500 ms | 7 Ping / 7 valid Pong | Idle error and HTTP fallback |
| B | JSON text event {"type":"keepalive"} |
1,500 ms | 14 application heartbeats | Normal WS completion, no idle error, no HTTP fallback |
| C | WebSocket protocol Ping | 6,000 ms | 14 Ping / 14 valid Pong | Normal WS completion, no idle error, no HTTP fallback |
Measured from the synthetic response.created event:
- Case A: the HTTP fallback POST arrived after 1.5125 seconds.
- Case A: the last valid Pong was received after 1.4655 seconds, approximately 47 ms before fallback.
- Case B: WebSocket completion was sent after 3.2036 seconds.
- Case C: WebSocket completion was sent after 3.2052 seconds.
The largest response.create frame payload, including warmup, was 42,471 bytes in every case. This reproduction does not involve a large request or image payload.
All three CLI processes exited with code 0 because the fixture permits a successful HTTP fallback in case A. The demonstrated problem is the idle classification and abandonment of a live delayed WebSocket—not a CLI crash or failed synthetic final answer.
The exact CLI warning in case A was:
Falling back from WebSockets to HTTPS transport. stream disconnected before completion: idle timeout waiting for websocket
The fixture deliberately uses plain HTTP/WS on loopback. “HTTPS transport” above is the CLI's own generic warning text; the lab does not perform TLS interception or weaken certificate validation.
What the experiment proves—and what it does not
Proven by the actual binary: protocol-level liveness can continue while the upper response waiter expires. A valid Pong was observed immediately before the CLI abandoned the connection. An application-level heartbeat resets the effective wait, whereas protocol Ping/Pong does not. Changing only the idle threshold also changes the outcome.
Proven by the fixture controls: the synthetic application has a viable delayed completion; cases B and C deliver it normally. The failure in A does not require an OpenAI outage, proxy misconfiguration, image history, or a dead TCP path.
Not proven: that every production idle timeout is a false positive; that Ping/Pong alone proves a real model is making progress; or that simply raising every timeout is a complete fix. A transport can remain alive while an application genuinely stalls. That is exactly why these conditions need separate accounting and diagnostics.
The current waiter is effectively measuring forwarded response-message inactivity, not transport liveness and not necessarily meaningful model progress. If the intended contract requires application heartbeats during long reasoning, please state and enforce that contract end-to-end rather than leaving users with opaque repeated retries.
Source-level explanation in 0.153.4
model-provider-info/src/lib.rsdefinesDEFAULT_STREAM_IDLE_TIMEOUT_MS = 300_000. The production config inspected did not contain a timeout override.- In
responses_websocket.rs, the WebSocket pump answersPingwithPongand consumesPongframes. These control frames are not forwarded through the message channel used by the response reader. - The same file's
run_websocket_response_streamawaitstimeout(idle_timeout, ws_stream.next()). Expiry generates the exactidle timeout waiting for websocketerror seen in the production logs and case A. responses_retry.rshandles retry/fallback after such stream errors. A new sampling step can have a fresh retry budget; repeated1/5notices in a long task are not a total per-task failure counter.- In the inspected provider merge function, an existing built-in provider entry wins through
entry(key).or_insert(provider). Consequently, the ordinary[model_providers.openai]approach is not a reliable way to override this built-in provider's timeout. The custom-provider override used by the lab should not be advertised as an already-validated production ChatGPT workaround.
The code explains why an unknown application-level heartbeat can preserve the wait while a recent protocol Pong does not. The local three-case run verifies that this distinction is present in the installed release, rather than only inferred from source or copied from another user's report.
Expected behavior and requested urgent action
Please address both false-dead-stream detection and genuinely dead streams:
- Track transport liveness separately, with explicit Ping/Pong and a bounded failure deadline when transport health checks fail.
- Track application silence/progress separately for an acknowledged in-flight response. Do not label the transport dead solely because no forwarded application event arrived during a legitimate long reasoning interval.
- Ensure a documented application heartbeat contract during long inference, or implement an equivalent client/server mechanism that the correct clock observes.
- Retain a bounded way to detect a truly stuck application; merely accepting Ping forever is not a complete solution.
- Expose supported timeout/retry/transport controls for the built-in ChatGPT provider, with effective values visible to users. Do not suggest a configuration key that the merge path ignores.
- Avoid repeatedly restarting a still-viable sampling operation without explaining what timed out. Show whether the request is acknowledged, when transport activity last occurred, and when application progress last occurred.
- Provide a supported mitigation for existing affected sessions and identify the release that will contain the fix.
Simply shortening the current timer risks interrupting more healthy long responses; simply lengthening it delays detection of genuine failures. The three-case result demonstrates why one undifferentiated inactivity clock is insufficient for these different states.
Please treat this as a severe, workflow-disrupting reliability issue and respond as soon as possible with an actionable mitigation and fix plan. I am not asserting data destruction or a universal failure of all prompts; I am reporting repeated real disruption plus a reproducible client behavior that can explain it.
Related issues and scope
- #39771: directly related high-reasoning false-dead-stream report. This adds an isolated local three-case reproduction on Windows stable 0.153.4.
- #38638: discusses slow detection of dead streams. A fix must also avoid terminating live, application-silent responses prematurely.
- #27625: repeated idle timeouts during one task.
- #23807 and #33051: related five-minute waits; not all such reports establish the same cause.
- #43015: my separate large-image-history report. The present local reproduction is image-free, and the inspected production session has only a small old image.
Diagnostic and privacy boundaries
No new full codex doctor --json report was collected. The investigation used targeted read-only production inspection, pinned source review, and the isolated actual-CLI test below. No production session was modified or interrupted by the test. Its temporary homes were separately verified and removed afterward; the reproduction source and results were preserved.
Raw production prompts, source code, images, session IDs, credentials, private request IDs and proxy account details are omitted. The code below uses only synthetic data and a local endpoint.
Reproduction instructions
Requires Python's standard library and an installed Codex CLI 0.153.4 executable. No additional Python packages or real API credentials are needed.
Save the code below as codex_ws_idle_repro.py, then run:
python -B .\codex_ws_idle_repro.py --codex 'C:\path\to\codex.exe' --output .\ws-idle-results.jsonIt runs three bounded cases, writes JSON results, and prints its uniquely created temporary directory for inspection and separate cleanup. It does not change the normal Codex home. It uses an isolated unauthenticated custom provider to redirect model requests to localhost; this differs intentionally from production authentication and timing.
The source below is the exact tested script, SHA-256 DB9ADD011C78D06624F1D404F0CB012969CDD78A6A52E8C02689999F723FD8C6.
Complete tested reproducer (Python standard library)
"""Exercise an installed Codex CLI against a loopback-only synthetic peer.
No OpenAI credentials, real model requests, project tools, or image data are used.
The caller retains the scratch directory for separate, verified cleanup.
"""
import argparse
import base64
import hashlib
import http.server
import json
import os
from pathlib import Path
import select
import socket
import struct
import subprocess
import tempfile
import threading
import time
class Case:
def __init__(self, name, heartbeat, idle_ms, delay_s=3.2):
self.name = name
self.heartbeat = heartbeat
self.idle_ms = idle_ms
self.delay_s = delay_s
self.events = []
self.lock = threading.Lock()
self.stop = threading.Event()
self.start = time.monotonic()
def log(self, event, **fields):
with self.lock:
self.events.append({"t_s": round(time.monotonic() - self.start, 4),
"event": event, **fields})
def recv_exact(sock, count):
chunks = bytearray()
while len(chunks) < count:
part = sock.recv(count - len(chunks))
if not part:
raise EOFError("peer closed")
chunks.extend(part)
return bytes(chunks)
def recv_frame(sock):
first, second = recv_exact(sock, 2)
opcode = first & 15
size = second & 127
if size == 126:
size = struct.unpack("!H", recv_exact(sock, 2))[0]
elif size == 127:
size = struct.unpack("!Q", recv_exact(sock, 8))[0]
if size > 2_000_000:
raise ValueError("synthetic-test frame limit exceeded")
mask = recv_exact(sock, 4) if second & 128 else None
payload = recv_exact(sock, size)
if mask:
payload = bytes(value ^ mask[i % 4] for i, value in enumerate(payload))
return opcode, payload
def send_frame(sock, opcode, payload):
size = len(payload)
if size < 126:
header = bytes([128 | opcode, size])
elif size < 65536:
header = bytes([128 | opcode, 126]) + struct.pack("!H", size)
else:
header = bytes([128 | opcode, 127]) + struct.pack("!Q", size)
sock.sendall(header + payload)
def response_events(response_id, warmup=False):
message = {"id": "msg_synthetic", "type": "message", "role": "assistant",
"status": "completed", "content": [{"type": "output_text", "text": "OK", "annotations": []}]}
response = {"id": response_id, "object": "response", "status": "completed",
"model": "gpt-6-astra", "output": [] if warmup else [message],
"usage": {"input_tokens": 1, "output_tokens": 0 if warmup else 1,
"total_tokens": 1 if warmup else 2,
"input_tokens_details": {"cached_tokens": 0},
"output_tokens_details": {"reasoning_tokens": 0}}}
if not warmup:
yield {"type": "response.output_item.added", "output_index": 0,
"item": {**message, "status": "in_progress", "content": []}}
yield {"type": "response.content_part.added", "item_id": "msg_synthetic",
"output_index": 0, "content_index": 0,
"part": {"type": "output_text", "text": "", "annotations": []}}
yield {"type": "response.output_text.delta", "item_id": "msg_synthetic",
"output_index": 0, "content_index": 0, "delta": "OK"}
yield {"type": "response.output_text.done", "item_id": "msg_synthetic",
"output_index": 0, "content_index": 0, "text": "OK"}
yield {"type": "response.content_part.done", "item_id": "msg_synthetic",
"output_index": 0, "content_index": 0, "part": message["content"][0]}
yield {"type": "response.output_item.done", "output_index": 0, "item": message}
yield {"type": "response.completed", "response": response}
class Peer(http.server.BaseHTTPRequestHandler):
protocol_version = "HTTP/1.1"
def log_message(self, *args):
pass
def do_GET(self):
case = self.server.case
if self.headers.get("Upgrade", "").lower() != "websocket":
case.log("non_websocket_get", path=self.path)
body = b'{"models":[]}'
self.send_response(200)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
return
key = self.headers.get("Sec-WebSocket-Key", "")
accept = base64.b64encode(hashlib.sha1((key + "258EAFA5-E914-47DA-95CA-C5AB0DC85B11").encode()).digest()).decode()
self.send_response(101)
self.send_header("Upgrade", "websocket")
self.send_header("Connection", "Upgrade")
self.send_header("Sec-WebSocket-Accept", accept)
self.end_headers()
self.wfile.flush()
self.close_connection = True
self.connection.settimeout(2)
case.log("websocket_upgraded")
try:
while not case.stop.is_set():
opcode, payload = recv_frame(self.connection)
if opcode == 8:
case.log("client_close")
return
if opcode == 10:
case.log("pong_received")
continue
if opcode != 1:
case.log("unexpected_client_opcode", opcode=opcode)
continue
request = json.loads(payload)
if request.get("type") != "response.create":
case.log("other_client_event", kind=request.get("type"))
continue
warmup = request.get("generate") is False
case.log("response_create_received", warmup=warmup, request_bytes=len(payload))
response_id = "resp_warmup" if warmup else "resp_synthetic"
created = {"type": "response.created", "response": {"id": response_id, "status": "in_progress", "model": "gpt-6-astra", "output": []}}
self.send_json(created)
case.log("response_created_sent", warmup=warmup)
if not warmup:
start = time.monotonic()
next_beat = start + 0.15
while time.monotonic() - start < case.delay_s and not case.stop.is_set():
now = time.monotonic()
if now >= next_beat:
if case.heartbeat == "protocol_ping":
send_frame(self.connection, 9, b"synthetic-liveness")
case.log("ping_sent")
else:
self.send_json({"type": "keepalive"})
case.log("application_keepalive_sent")
next_beat = now + 0.2
ready, _, _ = select.select([self.connection], [], [], 0.03)
if ready:
op, data = recv_frame(self.connection)
if op == 10:
case.log("pong_received")
elif op == 8:
case.log("client_close_during_wait")
return
elif op == 9:
send_frame(self.connection, 10, data)
case.log("client_ping_received")
else:
case.log("unexpected_during_wait", opcode=op)
if case.stop.is_set():
return
for event in response_events(response_id, warmup):
self.send_json(event)
case.log("completion_sent", transport="websocket", warmup=warmup)
except (EOFError, OSError, ValueError) as error:
case.log("peer_connection_end", error_type=type(error).__name__)
def send_json(self, event):
send_frame(self.connection, 1, json.dumps(event, separators=(",", ":")).encode())
def do_POST(self):
case = self.server.case
size = int(self.headers.get("Content-Length", "0"))
if size > 2_000_000:
self.send_error(413)
return
self.rfile.read(size)
case.log("http_fallback_post", request_bytes=size)
events = [{"type": "response.created", "response": {"id": "resp_http", "status": "in_progress", "output": []}}]
events.extend(response_events("resp_http"))
body = b"".join(b"data: " + json.dumps(event, separators=(",", ":")).encode() + b"\n\n" for event in events)
self.send_response(200)
self.send_header("Content-Type", "text/event-stream")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
self.wfile.flush()
case.log("completion_sent", transport="http", warmup=False)
def run_case(executable, scratch, name, heartbeat, idle_ms):
case = Case(name, heartbeat, idle_ms)
home = scratch / name / "home"
work = scratch / name / "work"
home.mkdir(parents=True)
work.mkdir()
server = http.server.ThreadingHTTPServer(("127.0.0.1", 0), Peer)
server.daemon_threads = True
server.case = case
threading.Thread(target=server.serve_forever, daemon=True).start()
port = server.server_address[1]
config = f'''model = "gpt-6-astra"
model_provider = "local_idle_probe"
approval_policy = "never"
web_search = "disabled"
[features]
memories = false
plugins = false
shell_snapshot = false
[memories]
use_memories = false
generate_memories = false
[model_providers.local_idle_probe]
name = "Loopback synthetic idle probe"
base_url = "http://127.0.0.1:{port}/v1"
wire_api = "responses"
requires_openai_auth = false
supports_websockets = true
stream_idle_timeout_ms = {idle_ms}
stream_max_retries = 0
request_max_retries = 0
'''
(home / "config.toml").write_text(config, encoding="utf-8")
allowed = {"SYSTEMROOT", "WINDIR", "COMSPEC", "PATHEXT", "PATH", "TEMP", "TMP", "USERPROFILE", "APPDATA", "LOCALAPPDATA", "HOMEDRIVE", "HOMEPATH", "NUMBER_OF_PROCESSORS", "PROCESSOR_ARCHITECTURE"}
env = {k: v for k, v in os.environ.items() if k.upper() in allowed}
env.update(CODEX_HOME=str(home), NO_PROXY="127.0.0.1,localhost,::1")
command = [str(executable), "exec", "--json", "--ephemeral", "--skip-git-repo-check", "--ignore-rules", "--color", "never", "--sandbox", "read-only", "--cd", str(work), "Reply with OK. Do not call any tools."]
start = time.monotonic()
timed_out = False
try:
result = subprocess.run(command, env=env, capture_output=True, text=True, encoding="utf-8", errors="replace", timeout=25, creationflags=getattr(subprocess, "CREATE_NO_WINDOW", 0))
stdout, stderr, code = result.stdout, result.stderr, result.returncode
except subprocess.TimeoutExpired as error:
timed_out = True
stdout = error.stdout.decode("utf-8", "replace") if isinstance(error.stdout, bytes) else error.stdout or ""
stderr = error.stderr.decode("utf-8", "replace") if isinstance(error.stderr, bytes) else error.stderr or ""
code = None
finally:
case.stop.set()
server.shutdown()
server.server_close()
combined = stdout + "\n" + stderr
counters = {}
for event in case.events:
counters[event["event"]] = counters.get(event["event"], 0) + 1
errors = [line for line in combined.splitlines() if "idle timeout" in line or "falling back" in line.lower() or '"type":"error"' in line]
errors = [line.replace(str(home), "<isolated-home>").replace(str(work), "<isolated-work>") for line in errors]
return {"name": name, "heartbeat": heartbeat, "configured_idle_ms": idle_ms,
"synthetic_completion_delay_s": case.delay_s, "process_timeout": timed_out,
"exit_code": code, "elapsed_s": round(time.monotonic() - start, 3),
"idle_timeout_reported": "idle timeout waiting for websocket" in combined,
"http_fallback_observed": counters.get("http_fallback_post", 0) > 0,
"event_counts": counters, "selected_client_messages": errors,
"events": case.events, "client_stdout": stdout.replace(str(home), "<isolated-home>").replace(str(work), "<isolated-work>"),
"client_stderr": stderr.replace(str(home), "<isolated-home>").replace(str(work), "<isolated-work>")}
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--codex", type=Path, required=True)
parser.add_argument("--output", type=Path, required=True)
args = parser.parse_args()
scratch = Path(tempfile.mkdtemp(prefix="codex-ws-idle-probe-"))
print(json.dumps({"scratch_directory": str(scratch)}), flush=True)
report = {"purpose": "Installed CLI versus loopback-only synthetic peer", "real_model_requests": 0,
"production_configuration_modified": False,
"codex_binary_sha256": hashlib.sha256(args.codex.read_bytes()).hexdigest(),
"cases": []}
for spec in [("ping_short_idle", "protocol_ping", 1500),
("application_heartbeat_short_idle", "application_keepalive", 1500),
("ping_long_idle", "protocol_ping", 6000)]:
result = run_case(args.codex, scratch, *spec)
report["cases"].append(result)
args.output.write_text(json.dumps(report, indent=2, ensure_ascii=False), encoding="utf-8")
print(json.dumps({k: result[k] for k in ["name", "exit_code", "elapsed_s", "idle_timeout_reported", "http_fallback_observed", "event_counts"]}), flush=True)
print(json.dumps({"output": str(args.output), "scratch_directory": str(scratch)}), flush=True)
if __name__ == "__main__":
main()
Activity
FengYeeLx commentedon Sep 6, 2026
@jif-oai — apologies for the direct ping. I am mentioning you specifically because you have very recently modified the same Responses WebSocket implementation involved here (
codex-rs/codex-api/src/endpoint/responses_websocket.rs).I would appreciate urgent triage or routing to the current owner because this now has a deterministic reproduction against the actual installed Windows Codex CLI 0.153.4 binary.
The controlled result is narrowly reproducible:
idle timeout waiting for websocketand falls back.The reproduction uses localhost only, no OpenAI inference, no credentials, no image data, and no production proxy in the model-request path.
A new
codex doctor --jsondiagnostic was also collected in #43168. It reports the active ChatGPT provider endpoints as reachable and the Responses WebSocket handshake as successful (HTTP 101 Switching Protocols).Additional internal diagnostics were uploaded through
/feedback:01a0755c-377e-75a1-9085-6f6a34bda5cfI am not claiming that Ping/Pong proves the model is making application progress. The demonstrated problem is narrower: the current error and retry path treats forwarded application-event silence as
idle timeout waiting for websocketeven while transport-level liveness is demonstrably continuing.That distinction matters for long-reasoning turns, where prematurely abandoning a viable in-flight response can cause repeated multi-minute retry/fallback cycles.
Could you please triage this or route it to the current owner of the Responses WebSocket transport? A supported mitigation for affected users would also be very helpful while a proper fix is developed.
Related prior report: #39771.
Please help. @jif-oai
FengYeeLx commentedon Sep 7, 2026
Status update — 2026-09-07
I re-checked the current
mainbranch rather than only the pinned 0.153.4 source.As of current
mainHEAD5ecb3afd1bf405149e2159bfda50093b0c1b5fab, the core mechanism demonstrated by the localhost reproduction still appears to be present:WsStreamstill handlesMessage::Pingby replying withMessage::Pong, whileMessage::Pongis consumed without forwarding either control frame through the response-message channel.run_websocket_response_streamstill waits on:tokio::time::timeout(idle_timeout, ws_stream.next())and still maps expiry to:
idle timeout waiting for websocketTherefore, transport-level Ping/Pong activity still does not appear to reset the application-message idle waiter on current
main.This is a source-level status check, not a claim that I rebuilt and re-ran the production workload against
main. The deterministic installed-binary reproduction in the original report remains the runtime evidence.The latest stable release is also still 0.153.4.
So, unless there is a server-side mitigation or an unmerged change that I cannot see, I do not currently see a client-side fix for the demonstrated mechanism in either the latest stable release or current
main.If a fix is already being developed elsewhere, please link or route this report to it. Otherwise, I would still appreciate maintainer triage, particularly on separating transport liveness from application-event silence for long-running Responses turns.
Related: #39771.
FengYeeLx commentedon Sep 8, 2026
Follow-up — September 8, 2026
Hi @jif-oai — my apologies for one further mention, and thank you for your time.
Could you please help route this report to the appropriate owner of Responses WebSocket timeout/heartbeat handling? I appreciate that the report was filed recently. Given the disruption documented above, I would be grateful for triage guidance or a supported mitigation; I am not asking for an immediate fix commitment.
A source-only check at
maincommit3caf9f9586baedb4158a7b91545ead3dd320c348still shows the relevant mechanism: the WebSocket pump handles Ping/Pong without forwarding those control frames to the response-message channel, while the response loop waits ontimeout(idle_timeout, ws_stream.next()).This is not a new runtime test against
main. The three-case localhost reproduction using the installed Windows 0.153.4 binary remains the runtime evidence: valid Pongs continue shortly before the idle timeout, while an application-level heartbeat allows the same delayed response to complete over WebSocket.I am not equating transport liveness with model progress, or suggesting that genuinely stuck responses should wait indefinitely. The point I would particularly appreciate clarification on is the intended heartbeat/timeout contract during an acknowledged, long-running response.
Is #39771 the preferred tracker for this behavior, or should this evidence be routed elsewhere? Is there a supported mitigation for affected users of the built-in OpenAI provider with ChatGPT sign-in?
The associated diagnostic reference is already recorded in #43168:
01a0755c-377e-75a1-9085-6f6a34bda5cfEven a brief pointer to the right owner or existing work would be very helpful. Please let me know which additional sanitized measurements would be most useful.
Thank you again for your time and consideration.
FengYeeLx commentedon Sep 8, 2026
@etraut-openai @bc-openai @riley-oai @rhan-oai @fcoury-oai @fchen-oai @felixxia-oai @jif-oai
I am deeply sorry for tagging so many of you at once.
I know this is intrusive, and I know that mass-mentioning maintainers is generally not an appropriate way to ask for help. I genuinely apologize for adding another notification to your workload. I would not do this under normal circumstances, and I do not intend to make a habit of it.
I am doing it only because I have run out of reasonable ways to determine how to get these two reports in front of the right person:
I have tried very hard to make both reports useful rather than simply reporting that “Codex is broken.” They include sanitized evidence, reproduction conditions, measurements, source-level observations, explicit limits on what the evidence does and does not prove, and suggested areas to investigate.
I also submitted an internal
/feedbackdiagnostic bundle and created #43168 solely so that the diagnostic thread could be associated with the two canonical reports.I understand that the Codex repository receives an extraordinary number of issues, that everyone here has their own responsibilities, and that many of you may have nothing to do with these particular components. I am not asking everyone I tagged to investigate the bugs, and I am absolutely not asking for an immediate fix or a release commitment.
I am asking for something much smaller:
If even one person here knows which engineer/team owns the relevant code, could you please help route these reports to them, or leave a brief indication of where I should take them?
Even something as short as:
or:
would be enormously helpful.
For #43022 in particular, I have tried to reduce the problem to a deterministic local transport experiment so that investigation does not depend on reproducing an intermittent production stall.
For #43015, I have tried to separate model-token accounting from aggregate serialized request size and to provide concrete byte measurements rather than speculation.
I am completely willing to provide additional sanitized diagnostics, run a narrowly specified experiment, test a candidate build, or correct anything in my analysis if I have misunderstood the implementation.
And if this mass mention itself is inappropriate, I sincerely apologize. Please feel completely free not to respond individually. I only need one route to the correct owner.
I care about Codex a great deal and use it heavily in real development work. My persistence here is not intended as criticism of any individual engineer. It comes from these failures having become serious workflow blockers for me, while I have been unable to determine any other reliable escalation path.
Again, I am very sorry for the broad notification, and I appreciate anyone who took the time even to read this far.
Thank you very much for your time, and please accept my apologies for the interruption.
doodead commentedon Sep 8, 2026
I don't condone such intrusive pinging, though I can see you're well aware of how it's not something good and you're only doing it because you perceive the situation as exceptional.
Additionally, though I wouldn't go that far, I, too, think this is a really disruptive issue and it needs to be fixed with urgency. Perhaps OpenAI already treats it with the urgency it deserves, in which case, I haven't really seen a lot of attempts on their part to communicate that. The impression I get is that it's being completely ignored, even as reports of the issue accumulate.
And just to emphasize: this issue really is quite disruptive for people with tasks that demand longer and more complex reasoning.
I especially echo this sentiment:
FengYeeLx commentedon Sep 8, 2026
Follow-up: consolidated related-issue map and evidence boundary
I am adding one final consolidated cross-reference map so maintainers do not have to rediscover the surrounding public reports individually.
I am not asserting that every issue below is a duplicate of #43022, nor that every production WebSocket failure has the same root cause.
The narrow result demonstrated by this issue remains:
That controlled result is intentionally narrower than the production symptom reports below.
1. Closest liveness / idle-watchdog reports
These are the most directly relevant reports to the distinction between transport liveness, application-event silence, and meaningful response progress.
#39771 — High-reasoning Responses WebSocket idle timeout is a false dead-stream
This is the closest prior production report. The same in-flight request eventually resumed after approximately 8.5 minutes when an application-level keepalive prevented the 300-second response-event idle watchdog from firing.
#42914 — Responses WebSocket black-hole after VPN disconnect leaves CLI Thinking for 5 minutes
This provides the important complementary failure mode: a genuinely black-holed transport can remain TCP-established and Codex waits approximately 300 seconds before the passive response-data idle timeout detects it.
Together, Responses WebSocket black-hole after VPN disconnect leaves CLI Thinking for 5 minutes #42914 and Severe WS stalls in 0.153.4: live Ping/Pong still trips the idle watchdog (Windows loopback-only reproduction) #43022 show why one undifferentiated timer cannot solve both problems.
#38638
Related dead-stream detection / idle-timeout behavior. It is important because simply increasing the existing timeout would make genuine failures slower to detect.
#27625
Repeated
idle timeout waiting for websocket/ reconnect behavior within long-running work.#23807 — codex-cli stalls for exactly 300s between tool result and next model request
Independent evidence around the exact 300-second boundary and repeated retry behavior.
#24260 — Codex Desktop: xhigh turn stalled 30m before first output, then resumed normally
Long application-silent interval followed by normal continuation.
#28579
Related long-running stream / idle-timeout behavior.
#26696
Related reconnect / retry behavior during long-running work.
#33051
Related multi-minute / five-minute response wait behavior.
#17003
Related stale/dead connection detection following connectivity changes.
The architectural conflict is therefore:
A single response-event inactivity timer cannot accurately represent both states.
2. Closely related Responses WebSocket disconnect / reconnect reports
The following issues do not all establish the specific Ping/Pong mechanism demonstrated in #43022, but they provide independent evidence around the same Responses streaming, retry, and recovery surface.
response.completed.These should be treated as symptom/corroboration reports, not automatically as duplicates of #43022.
3. Adjacent stream-state handling
#43140 — HTTP SSE response.failed waits for EOF and can be replaced by idle timeout
This is a separate deterministic
0.153.4client bug. Once the application has already delivered the terminalresponse.failedevent, the SSE consumer can continue waiting for transport EOF and later report an idle timeout instead of the original server error.It does not establish the same root cause as Severe WS stalls in 0.153.4: live Ping/Pong still trips the idle watchdog (Windows loopback-only reproduction) #43022, but it demonstrates the same broader architectural requirement: application state and transport lifetime should not be conflated.
4. Payload-driven WebSocket failures that can amplify the same user-visible symptom
These are deliberately separated from the core #43022 mechanism.
The controlled #43022 reproduction contains no images and the largest
response.createpayload is only approximately 42 KB, so large request history is not required to reproduce #43022.However, large-history failures can force retries/fallback and create longer periods of application silence, so they are relevant as production amplifiers.
#43015 — 63.8 MB image-history requests before any compaction
My separate report covering aggregate inline-image history and repeated large full-history requests.
#41338 — Inline image output costs ~230 tokens but 4.2 MB on the wire
Strong independent evidence for the byte/token mismatch.
#24550 — Responses WebSocket falls back when compacted replacement_history contains large inline images
Copied-session A/B evidence: removing only inline images from compacted replacement history eliminated the WebSocket retry/fallback behavior.
#38846 — Long CLI threads exceed Responses WebSocket message limit before auto-compaction, close code 1009
Particularly relevant because the peer supplied WebSocket close code
1009(Message Too Big) and Codex retried the unchanged request five times before HTTPS fallback.#39499 — Full-history fork replays compacted inline images and closes Responses WebSocket before response.completed
#40608 — Windows App image attachments trigger WebSocket 10054
#32555 — Image-heavy thread stays in Thinking and later surfaces WebSocket Broken pipe
#32533 — Responses Lite full-history resend after reasoning-effort change strands a large session until compaction
Again, these are not required to explain the isolated #43022 result. They demonstrate other routes through which production requests may spend more time in retry/fallback or application-silent states.
5. Diagnostic bridge for #43022 and #43015
This diagnostic upload is associated with the two reports and can be correlated with the previously supplied
/feedbackdiagnostic if useful.Why I believe the state model needs to be separated
The public reports now provide evidence for both opposite failure modes:
Case A: transport alive, application temporarily silent
#39771 and the controlled #43022 reproduction demonstrate that an acknowledged response may legitimately remain application-silent longer than the current response-event idle period while the connection remains viable.
Case B: transport actually dead, TCP/socket state not immediately reporting it
#42914 demonstrates that a black-holed path can leave an apparently established socket waiting for the full passive timeout.
Therefore:
A more robust state model would distinguish at least:
For example, a transport heartbeat/Pong deadline can detect an actually dead path without forcing a legitimate long-reasoning operation to emit visible model output every few minutes.
Conversely, successful Ping/Pong alone should not permit an application request to run forever; application progress can retain its own independent bounded policy.
Requested maintainer action
I would appreciate clarification on the intended liveness contract for the Responses WebSocket path and, if this behavior is confirmed:
I am intentionally not claiming that every linked issue has the same root cause. The purpose of this comment is to provide a complete cross-reference map around the liveness/retry problem while keeping the controlled claim of #43022 narrowly defined.
FengYeeLx commentedon Sep 8, 2026
@doodead Thank you — and I think your criticism of the mass ping is fair.
I knew it was intrusive when I did it, and I don't intend to repeat it. I had reached the point where I genuinely didn't know how else to get the reports routed, but that doesn't make broad pinging a good norm.
I also really appreciate you independently confirming the impact. The long-reasoning case is exactly where this becomes especially painful: a task can still be viable, yet several minutes of silence turn into another timeout/retry cycle and a great deal of lost time.
For fairness, OpenAI Support has since told me separately that the reports and diagnostics are under technical review. What I still don't know is whether that has reached the responsible engineering owner, and there has not yet been a public maintainer acknowledgement on the GitHub issues that I can see.
Hopefully the evidence and cross-references are sufficient now. I'll leave the routing to them rather than keep pinging people.
Thanks again for speaking up — having an independent user confirm the real workflow impact is genuinely helpful.
FengYeeLx commentedon Sep 9, 2026
Follow-up: the same idle-clock mechanism remains on current main, with a new independent production report
I re-checked the current public source after filing this issue. At main commit 73a1148, the relevant structure is still unchanged:
WsStream handles protocol Ping internally by replying with Pong;
received Pong frames are consumed internally;
only application-facing frames are forwarded through rx_message;
run_websocket_response_stream still waits with:
tokio::time::timeout(idle_timeout, ws_stream.next())
and reports:
idle timeout waiting for websocket
So protocol-level Ping/Pong activity still does not directly refresh the application-message idle clock exercised by the deterministic A/B/C reproduction in this issue.
There is also a new independent production report, #44099, from macOS / Codex CLI 0.153.4 / ChatGPT Pro, explicitly with no VPN or proxy. It reports repeated mid-turn WebSocket disconnects, five reconnect attempts, and HTTPS fallback; the reporter also notes that image-reading turns appear to make the symptom more frequent.
I am not claiming #44099 proves the same root cause, nor that every production WebSocket stall is caused by this idle-clock behavior. It is useful independent evidence that the broader failure class is not Windows- or proxy-specific.
The narrow regression requirement here remains the same: transport liveness, application-event activity, and semantic/model progress should be tracked as distinct signals rather than being collapsed into one application-message idle timeout.
FengYeeLx commentedon Sep 12, 2026
Help, please help!!
@etraut-openai
zentuit commentedon Sep 19, 2026
We are seeing the same issue across our organization. Our developers commonly run multiple Codex instances to multitask. When returning to an instance, we cannot tell whether Codex is still working or stuck on a dead WebSocket. Pressing escape and retrying the prompt clears the issue, but until we intervene the instance can sit there for the full five-minute timeout.
This uncertainty is disrupting normal workflows and frustrating our developers. We would appreciate a supported workaround and an update on whether a fix is being prioritized.
zentuit commentedon Sep 21, 2026
One difference however is that our development boxes are Linux based. It's not just a Windows issue