app-server loads ALL session files on every thread/list call — causes high CPU, slow startup, and background token waste #22411

Activity

1. etraut-openai

   added

   enhancementNew feature or requestNew feature or request

   app-serverIssues involving app server protocol or interfacesIssues involving app server protocol or interfaces

   performance

   on May 13, 2026
2. arkh-node commented on May 13, 2026

   

   arkh-node

   on May 13, 2026

   Author

   More actions

   Additional environment detail:

   The affected setup is the official VS Code extension:

   • Extension identifier: openai.chatgpt
   • Name: Codex – OpenAI’s coding agent
   • Version: 26.42.71525
   • Auto update: disabled
   • Running in a remote SSH workspace, so the app-server/session files are on the server side, not on my local laptop.

   A bounded check of current openai/codex main suggests that one inspected thread/list path is already metadata/head-only. So this may be specific to the extension/app-server version above, triggered by the SSH remote extension setup, or caused by another app-server path used during sidebar refresh/reconnect.

   The practical workaround that removed the high CPU behavior was archiving old ~/.codex/sessions files on the remote server and keeping only a few recent sessions.

   I’m happy to provide sanitized file counts/sizes and logs after I rest.
3. kangminlee-maker commented on May 26, 2026

   

   kangminlee-maker

   on May 26, 2026

   More actions

   I opened a separate issue with additional local evidence that may overlap with this all-session/list loading path: #24510. It includes state_5.sqlite active metadata size, the observed thread-list query/index plan, logs_2.sqlite TRACE/log evidence, long-running goal-session rollout stats, and a temporary mitigation script gist.
4. rnul commented on Jul 14, 2026

   

   rnul

   on Jul 14, 2026 – with ChatGPT Codex Connector

   More actions

   I can confirm this issue on a current Codex Desktop → Remote SSH → Linux app-server → Android remote-control setup, with a much larger session population and a clean before/after remediation.

   Environment

   • Codex Desktop client observed by app-server: 26.707.61608
   • Linux app-server runtime: 0.144.1
   • Linux PATH CLI: 0.142.3
   • Remote client: codex_chatgpt_android_remote (API v2)
   • Transport: ChatGPT remote-control relay to a Unix-socket app-server on Linux

   Reproduction and timing

   The Linux state store had:

   • 49,133 active threads matching source='exec' AND has_user_event=0
   • 3,215,116,658 bytes of rollout JSONL for those threads
   • no missing, duplicate, or out-of-tree rollout paths
   • healthy state_5.sqlite

   During one Android connection attempt, app-server logs recorded five consecutive thread/list requests on new remote-control connections. They took 34s, 36s, 35s, 39s, and 41s before Android retried. Every resumed rollout reported parse errors: 0.

   A direct app-server protocol comparison on the same profile showed:

   • default scan-and-repair thread/list: 45.77s first run, 39.17s second run
   • after compacting several large interactive rollouts: still 30.06s
   • thread/list with useStateDbOnly: true: 0.173s

   The generated protocol schema explicitly describes omitted/false useStateDbOnly as scanning JSONL rollouts to repair metadata. The Android request omitted this optimization.

   Controlled remediation

   I made a consistent SQLite backup and manifest, then reversibly archived only inactive rows matching:

   source = 'exec'
   AND has_user_event = 0
   AND archived = 0

   Rollouts updated during the preceding hour were excluded to avoid touching active automation.

   Results:

   • 49,090 inactive no-user exec threads archived
   • 3,212,078,641 bytes moved out of the active sessions tree
   • active JSONL count reduced to 246
   • 46 recent no-user exec sessions left active
   • current and backup SQLite databases both passed PRAGMA quick_check
   • every manifest source/destination/count/size reconciled

   Immediately afterward, the Android app connected and loaded all conversations successfully. No app-server restart, Linux reboot, account change, or network change was needed.

   Why automatic maintenance appears necessary

   Even after cleanup, 46 no-user exec sessions had been created or updated within the preceding hour. This profile will regress without retention/archival.

   Expected behavior would be one or more of:

   1. Completed no-user exec threads are automatically archived or excluded from interactive thread discovery.
   2. Remote/mobile thread/list uses the state-DB-only path and reconciles rollouts asynchronously.
   3. Filesystem repair work is bounded/incremental and cannot exceed the remote client's timeout.
   4. Thread retention/compaction prevents unbounded accumulation.

   I have not attached raw logs, rollout contents, account identifiers, hostnames, or local project paths, but can provide additional sanitized aggregate diagnostics if useful.

   👍React with 👍1Reacted by JM
5. MikeeI commented on Aug 3, 2026

   

   MikeeI

   on Aug 3, 2026

   More actions

   Hi, thanks for documenting this.

   I noticed one current downstream detail that may be relevant to the same rollout-scan root cause:

   • At audited upstream commit 5157493, latest_thread_cwd calls tokio::fs::read_to_string on the complete rollout and scans its lines in reverse to recover the latest TurnContext working directory.
   • In resolve_resume_thread_id, codex exec resume --last performs that read serially for each candidate returned by thread/list until the CWD matches.
   • The same path computes latest_thread_cwd before checking args.all, so resume --last --all still reads the first candidate rollout completely even though no CWD filtering is required.

   This appears to be a separate client-side full-file pass after thread/list. Improving the app-server listing and repair path alone may therefore leave codex exec resume --last reopening candidate rollout files to recover metadata that could potentially come from the state DB or thread projection.

   I have not measured the incremental latency of this second pass. The evidence above establishes only the current control flow and the fact that its work scales with the number and size of candidates inspected.

   Would it make sense for exec resume to short-circuit --all before calling latest_thread_cwd and otherwise consume an authoritative latest-CWD metadata projection instead of reopening each candidate rollout?

   I checked all relevant issues, comments, pull requests, discussions, and release notes; this evidence is not already reported.

   I am only reporting this finding and am not proposing a pull request unless a maintainer invites one.

   Disclosure

   Investigated thoroughly with GPT-5.6 Sol (runtime-default reasoning effort), using Oh My Pi as the agent framework.

   This report is not generic or unreviewed AI-generated output. Its claims were checked against the cited evidence, and it includes the relevant detail intended to help maintainers resolve the issue.

   If reports like this are not useful to the project, please let me know and I will refrain from submitting similar ones. My intent is to help without wasting maintainer time or energy or discouraging their work.

   Thank you for your work.
6. arkh-node commented on Aug 3, 2026

   

   arkh-node

   on Aug 3, 2026

   Author

   More actions

   Author of this issue here.

   I see #31305 ([thread-store] let stores own derived metadata) has been open since early July. If that work is paused, or if its scope is deliberately narrower than this issue, I'd rather take a small piece off your hands than duplicate anything.

   The part I have in mind is the one @MikeeI described above, which looks separable from the app-server listing path: latest_thread_cwd reads a whole rollout and scans it in reverse, resolve_resume_thread_id does that serially per candidate, and resume --last --all still pays for the first candidate even though no CWD filtering is needed.

   Two things I'd want to be honest about before anyone spends time on this. I can't reproduce the scale reported in this thread — my own profile is small, so the evidence above is rnul's and MikeeI's, not mine. And the change I'm describing is control-flow you can read and cover with a test, not something I could validate under real load.

   I'd only start if a maintainer says it's useful and not already in flight. Otherwise I'm happy to leave it be.
7. justinTM commented on Sep 3, 2026

   

   justinTM

   on Sep 3, 2026

   More actions

   This is a separate payload mitigation, not a fix for #22411's scan-and-repair bottleneck. Fork-only implementation proof: justinTM#1

   The short version: this fixes one measured sidebar payload amplifier, not the entire cold-open problem.

   • thread/list previews are bounded to 1,024 Unicode scalar values.
   • Full stored metadata, thread/read, history, names, and server-side search remain intact.
   • In the 2026-09-03T20:03:15Z 1,623-row snapshot, the projected preview JSON falls from 3.72 MB to 0.97 MB raw and from 0.86 MB to 0.22 MB after gzip, about 74 to 75 percent less.
   • The new regression test covers Unicode boundaries, unchanged short and exact-limit values, full history, beyond-preview search, and long names.

   This should reduce deep or repeated sidebar pagination over a hotspot or remote SSH connection. It does not reduce scan-and-repair, coalesce duplicate requests, prioritize a selected read, bound history hydration, or prove native Desktop cold opens are now fast. I left the useStateDbOnly default unchanged because its fallback and stale-index semantics need a separate compatibility decision.

   The preview behavior is also directly related to #21211.

   Diagnostic evidence, throughput control, and caveats

   • On the historical diagnostic runtime (codex-cli 0.144.1), Desktop navigation was usually acknowledged in 4 to 11 seconds; one one-turn read invocation against the large diagnostic task returned no response for more than 70 seconds. That was not a renderer-ready measurement on current main.
   • A prior complete state-DB-only list traversal returned about 2.92 MB raw in 14 pages. During a separate UI mount or refresh burst, the SSH proxy wrote 7.33 MB in 15 seconds while app-server logged 33 thread/list request events. The trace did not prove whether those events were distinct pages or overlapping traversals, nor whether the selected read waited behind them.
   • A settled 120-second window had no Desktop-originated list events and much lower traffic, so the list burst is episodic.
   • Read-only snapshot at 2026-09-03T20:03:15Z: 1,623 active previews, 3,611,401 characters total, 521 over 1,024, maximum 77,294. No private text was emitted during measurement.
   • The current projection removes 644,182 compressed bytes from an all-active traversal. At the observed 2.926 Mbps hotspot-route aggregate throughput, that is about 1.76 seconds of idealized link service; at the 5.851 Mbps immediate-recovery average, about 0.88 seconds. This is a throughput control, not an end-to-end prediction.
   • The newest 25 rows in the current snapshot had no preview above the cap, so this patch does not materially shrink every first page.
   • SSH compression was independently validated as active. The field-only offline payload comparison uses gzip level 6 specifically to avoid treating raw JSON savings as wire-byte savings; it retains a 74.7% reduction but is not a wire capture.
   • During the controlled hotspot route, aggregate physical-underlay throughput was 2.926 Mbps versus a 5.851 Mbps immediate-recovery average and a 10.867 Mbps recovery peak. No sustained aggregate saturation was observed, but transient RTT, jitter, radio scheduling, queueing, and per-flow congestion remain possible.
   • Adjacent hotspot context was 0% loss and 91.5 ms average RTT. High RTT still penalizes sequential pagination.
   • Direct state lookup, cached rollout parsing, and a 1.69 MB full-history read were much faster than the minute-scale symptom in isolation. Whole-rollout size also did not predict native route acknowledgement. Client scheduling, repeated pagination, history hydration, and renderer work remain open.
   • The patch intentionally does not cap user-defined names. It also does not change thread/read, resume, stored metadata, state-DB selection, or scan-and-repair.
   • Focused regression, clippy/fix, and formatting passed. The full app-server suite completed with 1,445 passed, 32 unrelated harness or fixture failures, and 2 skipped; none of those failures involved thread listing or previews.
   • The fork proof is based on 8f31b64c7f9ef67d8f966bff5ddf9e08eafe0b4d; upstream main had advanced to 32c303c197e437cb13d444389d651fbaae02a6ed after validation, so it is not presented as a current-upstream rebase.
   • No titles, prompt text, credentials, raw transcripts, or private paths are included.