Repository navigation
[Windows Desktop 26.901.6511.0] Five tasks stall; control RPCs take 15–27 minutes and shared request queue saturates #43525
Description
What version of the Codex App are you using?
- Windows package: 26.901.6511.0.
- Bundled backend executable: codex-cli 0.153.4, verified with that executable's
--version. - Local desktop-to-app-server transport: stdio.
- Configured default model:
gpt-6-astra, reasoning effortultra; individual task overrides were not exhaustively audited.
What subscription do you have?
ChatGPT Pro. At the post-incident check, the Codex usage window was 52% used and did not report a rate-limit restriction.
What platform is your computer?
Windows 11 Pro x64, version 10.0.26100, build 26100, approximately 64 GB RAM.
Five user-visible tasks were active across multiple local projects. The installation also had MCP/extensions, collaboration activity and scheduled heartbeat activity. The exact number of descendant tasks and all feature interactions have not been isolated.
Network environment: FlClash system proxy on localhost and a TUN/virtual network adapter. The user reported that the selected node appeared healthy. No contemporaneous proxy connection log or controlled direct-connection comparison was available. The proxy must therefore remain an environmental variable, not a proven cause or a proven non-factor.
What issue are you seeing?
Five active tasks appeared stuck for roughly an hour. Retained logs show intermittent progress and repeated shared control-path stalls, rather than proving that every task executed nothing for the entire hour.
The strongest evidence is that ordinary desktop control RPCs took 15–27 minutes to return, the shared client request pool reached six in-flight requests, and a queue snapshot contained 54 additional queued requests. Follow-up/steering submissions failed with:
App server request expired while queued
An app restart at approximately 2026-09-07 16:55 UTC / 2026-09-08 00:55 UTC+8 was followed by actual new task output. One task completed its current turn at 16:59:22 UTC; the other four produced fresh progress/tool results around 17:01 UTC.
This report concerns the shared control-request lifecycle and recovery behavior. The investigation found a concrete saturation mechanism in the installed client, but did not identify the first internal operation that started waiting or establish a full end-to-end root cause.
Actual impact
- Several independent tasks became difficult to observe or control at the same time.
- Task, account and configuration operations returned extremely late or expired in the queue.
- Critical
turn/steerrequests failed to submit. - Task status could still be
active/inProgresswithout a surfaced error; that status alone did not establish useful progress. - Application restart was followed by recovery without a configuration change performed by the investigator.
No task data loss was established. No in-app diagnostic feedback upload has been submitted as part of this report.
Measured control-request delays
These are the literal durationMs and queueWaitMs fields from desktop response_routed records, not estimates from a spinner. All response timestamps below are 2026-09-07 UTC.
| Response time | Method | durationMs | Approx. duration | queueWaitMs |
|---|---|---|---|---|
| 16:40:50.416 | getAuthStatus | 1,619,480 | 27.0 min | 0 |
| 16:40:50.430 | account/read | 1,617,618 | 27.0 min | 516 |
| 16:40:50.415 | config/read | 1,431,253 | 23.9 min | 21,286 |
| 16:39:25.990 | config/read | 1,346,868 | 22.4 min | 21,246 |
| 16:33:55.903 | turn/start | 1,046,790 | 17.4 min | 0 |
| 16:29:25.819 | mcpServerStatus/list | 936,514 | 15.6 min | 0 |
| 16:29:25.819 | account/read | 934,883 | 15.6 min | 0 |
These latencies concern control operations, so model reasoning effort alone does not explain them. The early backend TRACE records were no longer available for request/response pairing; these measurements do not, by themselves, distinguish time inside backend handlers from delayed response delivery.
Incident timeline
All timestamps in this table are 2026-09-07 UTC. Add eight hours for local time on September 8.
| Time | Observed event |
|---|---|
| 16:01:14 | Earliest queue rejection in the inspected retained desktop segment; six in-flight requests. |
| 16:07:10–16:07:12 | Five main task streams reported TLS disconnects. Successful reconnects followed within roughly 1–6 seconds. |
| 16:17:30–16:26:25 | Two task transcripts showed the same approximately 8.9-minute event gap. This is not proof of total backend inactivity. |
| 16:29–16:40 | Several very late control RPC responses arrived, including the 15–27-minute entries above. |
| 16:42:50.416 / 16:45:50.446 | Backend received two configRequirements/read requests; matching desktop responses followed approximately 95.648 / 185.706 seconds later. |
| 16:44:52 / 16:54:32 | Model refresh logged a child-process timeout. |
| 16:54:16.351 | Queue snapshot: 6 in-flight, 14 background queued, 9 critical queued, 31 interactive queued. |
| 16:54:20.746 | A steering submission failed because its request expired while queued. |
| 16:55:05 | Old app-server transport stopped; fresh app/backend processes started immediately afterward. |
| 16:59:22–17:01:33 | One current turn completed; all four other tasks had new actual output. |
Sanitized desktop-log excerpts
Only relevant fields are retained; request IDs, conversation IDs, project paths and unrelated fields are omitted. These are field-preserving excerpts, not complete raw lines.
2026-09-07T16:01:14.209Z warning [AppServerConnection]
app_server_client_request_queue_rejected
inFlightRequestCount=6 method=model/list priority=critical
queueWaitMs=3956 reason=queue-expired source=models
2026-09-07T16:40:50.416Z info [AppServerConnection] response_routed
durationMs=1619480 method=getAuthStatus queueWaitMs=0 errorCode=null
2026-09-07T16:40:50.430Z info [AppServerConnection] response_routed
durationMs=1617618 method=account/read queueWaitMs=516 errorCode=null
2026-09-07T16:54:16.351Z warning [AppServerConnection]
app_server_client_request_queue_rejected
inFlightRequestCount=6 method=config/read priority=background
queuedBackgroundRequestCount=14 queuedCriticalRequestCount=9
queuedInteractiveRequestCount=31 queueWaitMs=30008
reason=queue-expired source=config
2026-09-07T16:54:20.746Z error [electron-message-handler]
Error submitting steering turn for conversation
errorMessage="App server request expired while queued"
2026-09-07T16:55:05.568Z info [AppServerConnection]
Stopping app-server transport connectionId=1 transport=stdio
Additional backend signals
Earlier read-only inspection captured:
stream disconnected before completion:
IO error: peer closed connection without sending TLS close_notify
failed to refresh available models:
timeout waiting for child process to exit
The model-refresh timeout was reported at models-manager/src/manager.rs:349. The log did not identify the child command, child PID or wait stack. It is a local investigation lead, not proof that model discovery caused all the stalls.
After restart, there were also history-projection ordinal warnings and seven Custom tool call output is missing errors for one task between 16:55:23 and 17:00:10 UTC. That task subsequently produced actual progress. These are possible recovery/state-consistency symptoms; their causal relationship to the queue incident is unconfirmed.
What steps can reproduce the bug?
Original application incident: intermittent, initial trigger unknown
- Run several normal tasks across local projects in this Windows desktop build.
- Continue working while the app performs normal metadata, history and scheduled-task activity.
- In the affected session, control operations became very slow, all six shared in-flight slots were occupied, and queued requests began expiring.
- Attempt to steer an active task; the submission expires while queued.
- Restart the desktop app; actual task output subsequently resumes.
This is an observed incident sequence, not a deterministic minimal application reproduction. Opening Plugins, a particular prompt, a node switch, or a particular task history has not been shown to be the trigger.
Installed-client code inspection
Read-only inspection of app.asar/.vite/build/src-VqXTPopo.js in this exact build found:
AppServerConnectionowns a sharedrequestScheduler.- The scheduler class is minified as
RK;SK=6is the total in-flight limit, andCK=5limits non-critical work. This is an RPC limit, not a supported-task-count limit. markSentToTransportmarks a dispatched request as sent.dropUnsentRequestreturns false for sent requests.- In the ordinary already-dispatched branch of
abandonDestination, abandoning/timing out the caller does not itself call the scheduler'scomplete. The slot can remain occupied until the response arrives. There are special paths forplugin/listand unknown-outcome handling. getNextRequestIndexfirst stops dispatch when total in-flight requests reach six; that check also affects newly queued critical work.
Retaining accounting for an operation with an unknown outcome may be intentional. The concern is whether long-lived sent requests have a bounded recovery path that preserves host responsiveness. Blindly freeing slots or replaying mutating requests would not necessarily be a safe fix.
Deterministic scheduler-only experiment
An isolated harness loaded the original scheduler code from the installed bundle. It used three background metadata requests, one background history request, one interactive account request, and one critical start request, all marked as sent and given no responses.
Results:
- All six occupied in-flight slots.
- All six
dropUnsentRequestattempts returned false. - A new critical
turn/interruptprobe expired in the queue; the original six slots remained occupied. - After simulating one old response with
complete, a new critical probe dispatched immediately.
{"releaseAttempts":[false,false,false,false,false,false],"afterProbeTimeout":{"inFlightRequestCount":6,"queuedBackgroundRequestCount":0,"queuedCriticalRequestCount":0,"queuedInteractiveRequestCount":0},"events":["queue-expired"]}
{"afterOneOldResponse":["queue-expired","recovery dispatched"],"passed":true}Bundle file SHA-256: 8d056524ea3f5714e5e0a254afd88f018d58465bc6ea026adee3c80ff7799e29.
The experiment verifies scheduler behavior under simulated non-response. It does not execute the full bridge timeout path, reproduce the initial backend delay, demonstrate a deployed end-to-end fix, or prove that the original six requests were exactly these synthetic methods.
Runnable, build-specific scheduler-only harness (Python 3 + Node.js)
Save as scheduler_repro.py and run python scheduler_repro.py "C:\path\to\app.asar". It reads the bundle and runs its scheduler in a separate Node process; it does not modify Codex, access tasks or make network requests.
"""Scheduler-only reproduction for Codex Desktop 26.901.6511.0.
Reads the installed ASAR without modifying it. Requires Python 3 and Node.js.
No network calls, credentials, task content, or running Codex processes are used.
Usage: python scheduler_repro.py "C:\path\to\app.asar"
This reproduces saturation behavior, not the original unknown backend trigger.
"""
import argparse
import hashlib
import json
from pathlib import Path
import struct
import subprocess
import sys
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("asar", type=Path)
args = parser.parse_args()
with args.asar.open("rb") as stream:
sizes = struct.unpack("<4I", stream.read(16))
if sizes[0] != 4 or sizes[3] > 20_000_000:
raise SystemExit("Unsupported ASAR header")
header = json.loads(stream.read(sizes[3]))
entry = header["files"][".vite"]["files"]["build"]["files"]["src-VqXTPopo.js"]
stream.seek(8 + sizes[1] + int(entry["offset"]))
source_bytes = stream.read(entry["size"])
source = source_bytes.decode("utf-8")
start = source.index("SK=6,CK=5,")
end = source.index("var VK=", start)
scheduler_source = "var " + source[start:end]
print(json.dumps({
"scope": "scheduler-only; original backend hang not reproduced",
"bundle_file": ".vite/build/src-VqXTPopo.js",
"bundle_sha256": hashlib.sha256(source_bytes).hexdigest(),
"scheduler_start_character": start,
"scheduler_end_character": end,
}), flush=True)
harness = r"""
const assert = require("node:assert/strict");
const scheduler = new RK();
const events = [];
const oldRequests = [
{method:"config/read", priority:"background", source:"config"},
{method:"model/list", priority:"background", source:"models"},
{method:"permissionProfile/list", priority:"background", source:"permissions"},
{method:"thread/read", priority:"background", source:"tail_history"},
{method:"getAuthStatus", priority:"interactive", source:"account"},
{method:"turn/start", priority:"critical", source:"turn"}
];
oldRequests.forEach((r, i) => {
const requestId = "old-" + i;
scheduler.enqueue({
requestId, method:r.method, params:{testOnly:i},
scheduling:{priority:r.priority, source:r.source},
dispatch:() => scheduler.markSentToTransport(requestId),
reject:result => { throw Error("Unexpected initial rejection: " + result.reason); }
});
});
assert.equal(scheduler.getQueueState().inFlightRequestCount, 6);
const releaseAttempts = oldRequests.map((_,i) => scheduler.dropUnsentRequest("old-"+i));
assert.deepEqual(releaseAttempts, [false,false,false,false,false,false]);
scheduler.enqueue({
requestId:"interrupt-probe", method:"turn/interrupt",
scheduling:{priority:"critical", timeoutMs:40},
dispatch:() => events.push("interrupt dispatched"),
reject:result => events.push(result.reason)
});
setTimeout(() => {
try {
assert.deepEqual(events, ["queue-expired"]);
assert.equal(scheduler.getQueueState().inFlightRequestCount, 6);
console.log(JSON.stringify({
releaseAttempts, afterProbeTimeout:scheduler.getQueueState(), events:[...events]
}));
// Simulate receiving one old request's response.
scheduler.complete("old-0");
scheduler.enqueue({
requestId:"recovery-probe", method:"turn/interrupt",
scheduling:{priority:"critical"},
dispatch:() => events.push("recovery dispatched"),
reject:result => events.push(result.reason)
});
assert.deepEqual(events, ["queue-expired", "recovery dispatched"]);
console.log(JSON.stringify({afterOneOldResponse:events, passed:true}));
} catch (error) {
console.error(error.message);
process.exitCode = 1;
} finally { scheduler.clear(); }
}, 120);
"""
result = subprocess.run(
["node", "-"], input=scheduler_source + harness,
text=True, capture_output=True, timeout=10,
)
print(result.stdout, end="")
if result.stderr:
print(result.stderr, file=sys.stderr, end="")
raise SystemExit(result.returncode)What is the expected behavior?
- Slow discovery, account, configuration or history operations should not leave all tasks on the same local host unresponsive for tens of minutes.
- Cancellation/recovery controls should remain usable or provide a bounded, explicit recovery action when the shared pool is saturated.
- Timeouts should distinguish “not sent” from “sent, outcome unknown,” with safe reconciliation instead of blind mutation retries.
- Background polling should be bounded/coalesced during saturation.
- Task state should expose when useful progress/control has stalled instead of relying only on active/inProgress.
- Recovery should preserve task history/tool-result consistency and should not ordinarily require restarting the entire desktop app.
Additional information
Is this caused by FlClash or the network?
The available evidence does not establish that FlClash caused this incident, and it also does not completely exonerate the network:
- The queue had already started rejecting requests before the observed 16:07 TLS burst.
- The observed WebSocket reconnects succeeded within seconds.
- Model-list HTTP requests continued to return 200 at several points during the affected period.
- Between 16:42 and 16:54 the backend continued receiving some requests and completing some HTTP calls; this was not demonstrated to be a complete process deadlock or a continuous network outage.
- A healthy node indicator is not a trace of every long-lived connection. No controlled proxy/direct-connection A/B test was performed.
- FlClash processes had been running since well before the incident. No contemporaneous proxy error log was available.
- Windows logs reviewed for the incident window contained no corresponding sleep/resume, application-hang/crash or resource-exhaustion event. Absence of those records does not rule out shorter stalls.
- Post-incident memory/disk availability was ample, but no continuous resource trace exists for the failure window.
The strongest current hypothesis is prolonged control-request non-completion, amplified by shared-slot retention and insufficient recovery/isolation. The initiating delay may be in a backend handler, child process, IPC/delivery path, or an interaction with network recovery. Please investigate both the first slow operation and the propagation across tasks.
Evidence limitations and requested diagnostics
Early backend TRACE rows were dynamically removed during inspection. Some previously read SQLite log row IDs no longer existed on a later query. Thus, the exact original six occupying requests and the initial waiting stack could not be reconstructed reliably. No complete transport trace or process dump was captured.
Useful engineering diagnostics would include:
- In-flight request method, age, priority, owner and sent/caller-timeout state at each saturation event.
- Correlated backend receive/start/finish/send timestamps and desktop receive/dispatch timestamps.
- The model-manager child command/PID and wait stack for the two child-exit timeouts.
- Whether account/config/history operations share a lock or another long-lived dependency.
- A bounded fault-injection test that holds backend replies while exercising metadata polling, task steering and cancellation.
- Tests for late responses, unknown mutation outcomes and history/tool-output consistency after recovery.
Related reports
- Windows Desktop: Opening Plugins saturates app-server queue and causes subsequent message sends to time out #37401 — similar Windows shared-queue starvation with a demonstrated Plugins trigger. This incident does not establish that trigger and is on 26.901.6511.0.
- [Windows Desktop] Selected recent task stays blank while collaboration hydration starves app-server queue #36914 — related shared-pool starvation during collaboration/history hydration. This report adds minute-scale account/config/start delays and the sent-request retention experiment.
- Windows Desktop 26.803: active turns show no progress for 3–4 hours while the app remains interactive #37775 — related multi-task visible stalls on an older Windows build.
These may share an underlying issue; I am not claiming a distinct confirmed root cause. Please consolidate if appropriate.
This public report contains selected diagnostic fields and a synthetic harness only. Task contents, full transcripts, local project paths, credentials, proxy subscriptions, and conversation/request identifiers are omitted.
Activity
github-actions commentedon Sep 7, 2026
Potential duplicates detected. Please review them and close your issue if it is a duplicate.
Powered by Codex Action
Kanturna commentedon Sep 8, 2026
Additional Windows incident evidence: global model picker/send lockout, September 6–8
I am seeing the same failure pattern on another Windows installation. The model selector disappears across all tasks, including new chats, and messages cannot be submitted. It sometimes recovers later; restarting the app also restores model-list responses.
Current verified environment: Windows desktop package 26.901.6511.0, bundled codex-cli 0.153.4, local
stdiotransport. MCP integrations and concurrent tasks are present. Earlier retained logs below include app sessions before the current restart.Local measurements (all timestamps UTC)
model/listwith six in-flight requests.model/listand fourturn/steer; observed queue snapshots reached 103 waiting requests. These are request counts, not independent incidents.account/readreturned after 1,261,504 ms andgetAuthStatusafter 1,261,517 ms, both withqueueWaitMs=0anderrorCode=null.turn/steerrequests were rejected.model/listexpired after 30 seconds in the shared queue.model/listrequests returned successfully within milliseconds.Queue totals sum the background, critical, and interactive queued-count fields; the separate priority-count field is not added again.
Field-preserving excerpts, with identifiers and unrelated fields omitted:
The backend also logged this at September 8, 16:22:58, 16:27:33, 16:32:08, and 16:55:29 UTC:
The child command/PID and its wait stack were not captured, so model refresh is an investigation lead, not a proven initiating cause.
Context-management and account-switching boundaries
The visible problem was first noticed around account switching within the same app and an attempted burst of roughly 19 tasks. This is an observed workload sequence, not a deterministic reproduction or proof that account switching/concurrency caused the problem; earlier retained logs already show the failure signature.
[features.context_management] experimental_mode = truewas added only on September 8 at approximately 04:43 UTC, after the September 6–7 failures. The pre-change backup parses successfully and lacks that section; the current config parses successfully with it. Thus this particular opt-in cannot explain the original onset. Model-list requests also recovered after restart while the flag remained enabled. This does not rule out a later interaction.Read-only inspection of the installed
.vite/build/src-VqXTPopo.jsproduced the same SHA-256 as the original report:8d056524ea3f5714e5e0a254afd88f018d58465bc6ea026adee3c80ff7799e29.No scheduler patch was applied, and I am not claiming to have reproduced the original backend stall with a synthetic harness.
The strongest local evidence is shared control-path saturation plus very late replies. There is no process dump or continuous resource/network trace from the stalls, so the first blocking operation remains unresolved. This also resembles #38385.
Please investigate bounded recovery for already-dispatched control requests and capture the method/age/wait state of the requests occupying the shared slots when model selection and message submission become unavailable.
tao19911024-gif commentedon Sep 11, 2026
Additional Windows evidence: shared control-request saturation and retained MCP processes (2026-09-11)
The following observations extend this report to desktop package 26.903.9818.0, with bundled codex-cli 0.153.4 on Windows x64. The desktop-to-backend transport was local stdio. The evidence supports a shared control-path responsiveness problem; it does not establish the operation that initiated the backend delay.
Observed incident
All timestamps below are UTC on September 11, 2026.
thread/startrequest reached its 30,000 ms caller timeout.durationMs=769177andqueueWaitMs=0, approximately 12 minutes 49 seconds after dispatch.app_server_client_request_queue_rejectedevents. This is an event count, not a count of separate incidents.Isolated scheduler reproduction
A foreground Node harness exercised the original scheduler extracted from the installed desktop bundle. The experiment used simulated requests and responses; it did not contact the running app-server, create real tasks, invoke a model, or modify the installation.
thread/startrequests, mark them as sent, and withhold their responses.turn/interruptfor a different synthetic task and a configuration read.dropUnsentRequest; verify that it returns false.Recorded results:
{ "sixDispatched": 6, "otherThreadInterruptQueued": 1, "configQueued": 1, "cannotDropAlreadySent": true, "interruptUnblockedOnlyAfterCompletion": true, "configUnblockedAfterAnotherCompletion": true }Bundle entry:
.vite/build/src-B6LqG3ek.jsBundle SHA-256:
9a1d9737c526cbba0e18e5ffc3b9371ef055318e7eabb7584b082fbcdb24472eThis reproduces scheduler behavior under non-completion. It does not exercise the complete caller-timeout path or reproduce the original backend wait. Retaining accounting for a sent operation may be necessary when its outcome is unknown; simply freeing slots or replaying mutations would not be a sufficient fix.
Post-restart MCP process retention
At approximately 07:52–07:54 UTC, the replacement backend had 235–236 OS threads. A live process snapshot contained ten groups of six direct children, created between 07:01 and 07:51 at roughly five-minute intervals, in addition to the initial group. Parent identity and process creation times were checked.
The installed webview contains an
mcp-extension-host-threadquery withstaleTime: Infinity, no descriptor-levelgcTime, and an imperativegetOrFetchpath. This is consistent with the mechanism discussed in #43122, but live observer state and cache-eviction events were not captured.The matching backend source also contains the Windows stdio path discussed in #42740. The observed OS thread count is not a measurement of blocking-pool occupancy, and the failed process had already exited before a waiting-stack capture was available. Process retention, historical thread-pool exhaustion, and the initiating delay must therefore remain distinct findings. Network involvement remains unresolved.
Suggested remediation and acceptance criteria
These are proposed investigation and design directions, not a validated patch:
The related merged #43870 addresses stderr-reader cleanup on teardown. Its scope does not establish a fix for still-retained sessions or shared-slot saturation.
This supplement includes only selected diagnostic fields and synthetic experiment results. Raw transcripts, credentials, cookies, private project paths, process command lines, and conversation/request identifiers have been omitted.
Miles-Cheverton commentedon Sep 14, 2026
Adding a recovery comparison from macOS report #45405. My Desktop logs also reached six in-flight requests, with unrelated configuration/model/thread requests expiring while queued. Local Work could not submit even the first turn of a fresh conversation under a clean
CODEX_HOME.On my Mac Studio, Allow Connections OFF in ChatGPT Desktop → wait a few seconds → ON restored Local Work after app/runtime restarts and user-MCP disabling had failed. No OS downgrade was needed.
Environment: ChatGPT 26.908.40834 (8881), bundled Codex 0.154.0-alpha.6.2, macOS 27.0 build 26A428, arm64.
I have not measured the 15–27-minute delays reported here, reproduced your scheduler experiment, or established what the toggle reset internally. This is a recovery observation for comparison; a shared root cause or Windows remedy remains unproven. Full details: #45405.
0xdevalias commentedon Oct 17, 2026
Related current macOS build-20398 evidence: #53197 (comment) documents the eight-total-active scheduler, separate queued caps, ten-renderer demand burst, critical rejections, and dispatched read-timeout retention. Includes hashes and tested offline scheduler/retry probes. The original local submission stall occurred outside the measured rejection bursts, so a common initiating cause is not asserted.