Skip to content

Windows Desktop: unbounded taskkill.exe/conhost.exe cleanup storm exhausts WMI #34260

Open

Description

@RocStone

Summary

Codex Desktop on Windows can enter an unbounded process-cleanup loop where hundreds of taskkill.exe processes remain alive, each with a corresponding conhost.exe. The accumulated taskkill instances repeatedly query Win32_Process, exhaust the WMI provider quota, and make the entire Windows UI sluggish or unable to launch new processes.

This is related to #17229 and #21957, but the leaking processes in this case are taskkill.exe themselves rather than only git.exe/conhost.exe or leaked stdout.

Environment

  • Codex Desktop: 26.715.4045.0
  • OS: Windows 25H2 x64, build 26200.8875
  • RAM: 32 GB
  • Installation: Microsoft Store/MSIX package OpenAI.Codex_26.715.4045.0_x64__2p2nqsd0c76g0

Observed behavior

During normal local-agent use involving shell commands, cancellations, and command timeouts, process counts grew continuously.

Peak measurements:

  • taskkill.exe: 457 processes
  • conhost.exe: 473 processes
  • total processes: 1,388
  • total threads: over 14,000
  • total handles: over 350,000
  • WMI provider private memory: 720.5 MB

All sampled taskkill.exe instances had the Codex Desktop main process (ChatGPT.exe inside the OpenAI.Codex package) as their parent. Most excess conhost.exe instances were children of those taskkill.exe processes.

Windows Event Viewer recorded:

  • WMI event 5612: PrivatePageCount exceeded its 512 MB quota.
  • WMI event 5612: ThreadCount exceeded its 256-thread quota.
  • WMI Activity event 5858: repeated Win32_Process queries failed with 0x80041006 and 0x80041033.
  • taskkill.exe crashed with exception 0xc0000005 (Application Error event 1000 / Windows Error Reporting event 1001).

Killing the accumulated taskkill.exe processes immediately reduced the total process count from 1,388 to about 565, and WMI private memory fell from 720.5 MB to about 104 MB. However, while the original Codex process remained running, taskkill.exe counts grew again from 156 to 320 in 20 seconds.

Fully restarting/reinstalling Codex cleared the bad state. With the same app version reinstalled, the stable baseline was:

  • taskkill.exe: 0
  • conhost.exe: 12
  • total processes: about 390
  • WMI: 139 MB private memory / 54 threads

Because the reinstalled version is identical, this appears to clear the accumulated state rather than fix the underlying bug.

Relevant packaged-app implementation

Inspection of the installed app.asar shows that the Windows process-tree cleanup path invokes approximately:

execFile(taskkill.exe, ["/pid", pid, "/t", "/f"], {
  windowsHide: true,
});

The call does not appear to set a timeout. Cleanup is deduplicated per execution object, but there is no apparent global concurrency limit, backoff, or circuit breaker across multiple local executions. If taskkill.exe hangs because process enumeration/WMI is unhealthy, later cleanups can start additional taskkill.exe instances indefinitely.

Steps that may reproduce

  1. Open Codex Desktop on Windows and a local repository.
  2. Run multiple local shell operations, including operations that time out or are cancelled.
  3. Observe taskkill.exe and conhost.exe in Task Manager or Process Explorer.
  4. If WMI/process enumeration becomes slow, observe whether taskkill.exe instances remain alive and increase continuously.
  5. Keep the app open and observe WMI event 5858 failures and growing process/thread/handle counts.

The trigger is intermittent; the unbounded accumulation after the first stuck cleanup is the main defect.

Expected behavior

  • A process-tree cleanup should complete or time out within a bounded period.
  • Codex should not run hundreds of concurrent taskkill.exe cleanup processes.
  • Repeated cleanup failures should activate backoff/circuit breaking and surface one actionable error to the user.
  • Failure of WMI/process enumeration should not cause a system-wide process storm.

Suggested fixes

  1. Add a timeout to the taskkill.exe invocation and forcibly terminate a stuck cleanup helper.
  2. Add a process-wide single-flight or small concurrency limit for Windows tree cleanup.
  3. Add exponential backoff/circuit breaking after repeated cleanup failures.
  4. Prefer Windows Job Objects or direct Win32 process-tree termination instead of spawning taskkill.exe for every cleanup.
  5. Log the target PID, duration, exit status, and number of concurrent cleanup operations.

Activity

  1. added
    bugSomething isn't working
    appIssues related to the Codex desktop app
    windows-osIssues related to Codex on Windows systems
    tool-callsIssues related to tool calling
    on Jul 20, 2026
  2. frank9306 commented on Jul 20, 2026

    @frank9306

    I'm also affected by this issue and would appreciate any updates on the progress. Thanks for the work on this.

  3. msg7086 commented on Jul 20, 2026

    @msg7086

    26.715.4045.0 here also affected by this. My work laptop is frozen with 80% of the CPU wasted on ChatGPT taskkills.

  4. askxue commented on Jul 20, 2026

    @askxue

    I have been suffering from this bug for several days now. I hope it can be fixed as soon as possible.

    Image
  5. adfras commented on Jul 20, 2026

    @adfras

    I can reproduce this on a newer Codex Desktop build, so the issue persists beyond 26.715.4045.0.

    Environment

    • Codex Desktop: 26.715.7063.0
    • Package: OpenAI.Codex_26.715.7063.0_x64__2p2nqsd0c76g0
    • Windows 11 Pro x64, build 10.0.26200
    • 32 GiB RAM

    Reproduction and evidence

    After fully quitting Codex, restarting Windows, checking for app updates, and relaunching Codex, the process storm returned.

    • Many live C:\Windows\System32\taskkill.exe /pid <pid> /t /f processes accumulated.
    • Every sampled taskkill.exe had the packaged Codex desktop main process (ChatGPT.exe) as its direct parent.
    • Task Manager showed approximately:
      • Service Host: Windows Management Instrumentation: 14.5% CPU
      • WMI Provider Host: 13.1% CPU
      • ChatGPT/Codex: 8.2% CPU
      • Total system CPU: 67%
    • Even a trivial one-line PowerShell command invoked through Codex timed out.
    • A direct git status --short --untracked-files=no completed in about 0.2 seconds, so repository Git responsiveness was not the trigger in this reproduction.

    The Codex desktop log repeatedly reports:

    [electron-sampler] Failed to collect child process snapshot
    powershell.exe ... Get-CimInstance Win32_Process ...
    

    Windows WMI Activity event 5858 records repeated Win32_Process query failures from the cleanup processes with result codes:

    • 0x80041032
    • 0x800706BE
    • 0x800706BA

    Inspection of this build's packaged app.asar also shows that the Electron sampler:

    • constructs the Windows child-process snapshot worker unconditionally,
    • requests a heartbeat snapshot every 30 seconds while the primary window is focused,
    • permits additional triggered snapshots with a 5-second minimum interval.

    This appears to amplify the failure: a timed-out WMI snapshot leads to taskkill cleanup; each stuck taskkill performs another Win32_Process query; later sampler cycles add more cleanup processes without effective global backoff/circuit breaking.

    The same sampler failure first appears in local Codex logs from 2026-07-15 under desktop build 26.707.9981.0, and continues under 26.715.7063.0. No private paths, repository names, prompts, or raw logs are included here. Sanitized screenshots and selected log/event excerpts are available if maintainers need them.

  6. levgoriachev commented on Jul 20, 2026

    @levgoriachev

    +1. Made whole app completly unuseable. With 2 220$ subscriptions. Great work, devs!
    win 10, 26.715.52143

  7. NocturnalSk commented on Jul 20, 2026

    @NocturnalSk

    Confirmed on a newer Microsoft Store build with another Windows 10 reproduction.

    Environment

    • Codex Desktop package: OpenAI.Codex_26.715.7063.0_x64__2p2nqsd0c76g0
    • Package status/signature: Ok / Store
    • Bundled ChatGPT.exe file version: 150.0.7871.124
    • OS: Windows 10 Pro 22H2 x64, build 19045.6466
    • Logical processors: 16
    • Observed: 2026-07-21 00:24:08 (Asia/Shanghai)

    Observed behavior and evidence

    Task Manager showed the following simultaneous CPU spike:

    • Service Host: Windows Management Instrumentation: 14.1%
    • WMI Provider Host: 11.8%

    A read-only process/event investigation found:

    • Winmgmt was hosted normally by C:\Windows\System32\svchost.exe -k netsvcs -p.

    • Windows svchost.exe, WmiPrvSE.exe, and taskkill.exe binaries had valid Microsoft signatures.

    • Before the screenshot, the WMI Activity log already contained 742 Event ID 5858 failures from 00:17 through 00:23. The operations repeatedly enumerated Win32_Process and Win32_PerfFormattedData_PerfProc_Process, with result codes including 0x800706BA, 0x80041032, and 0x80041033.

    • One live snapshot found 46 taskkill.exe processes, all direct children of the Codex Desktop main ChatGPT.exe process. Sampled command lines were of the form:

      C:\Windows\System32\taskkill.exe /pid <pid> /t /f
      
    • A later five-second sample started with 212 live taskkill.exe processes and ended with 202 still alive.

    • During one 15-second polling window, 1,092 distinct PIDs were observed (including the initial live set), showing extreme intermittent process churn.

    • A later five-second CPU sample showed Winmgmt and all four WmiPrvSE.exe instances back at 0%, consistent with a bursty query/cleanup storm rather than sustained load.

    • The WMI permanent-subscription check found no CommandLineEventConsumer or ActiveScriptEventConsumer; only the standard SCM event-log subscription was present. No TCP connections were owned by the WMI provider PIDs.

    Two later diagnostic commands timed out and may have amplified the subsequent taskkill.exe count. However, the 742 WMI failures and the Task Manager spike occurred before those diagnostics, so the underlying cleanup/query storm was already active.

    This matches the unbounded Windows cleanup behavior described in this issue and confirms that it still reproduces in Store build 26.715.7063.0 on Windows 10. A bounded timeout plus global concurrency limit/backoff around taskkill.exe would prevent this failure from escalating into system-wide WMI and process exhaustion.

    Usernames, repository paths, conversation contents, and unrelated process details were intentionally omitted. A screenshot of the WMI CPU spike can be provided if useful.

  8. noman-qadir commented on Jul 20, 2026

    @noman-qadir

    Additional WPR and controlled Git/MCP isolation evidence

    I have an independent reproduction on:

    • Codex Desktop/MSIX 26.715.4045.0
    • Windows 11 25H2, build 26200.8875, x64
    • Native Windows Git repositories, not WSL

    A 4.09 GiB Windows Performance Recorder trace captured the process storm with zero lost events or buffers. During its final 385-second incident window:

    • 2,000 genuine taskkill.exe launches occurred.
    • 1,986 were children of the main ChatGPT/Codex Desktop process.
    • 1,908 targeted git.exe.
    • Approximately 1,983 Git launches ran git config --null --get core.fsmonitor.
    • Peak taskkill concurrency was 33, with a peak creation rate of 12 per second.

    I later performed a controlled MCP-isolation comparison:

    1. Temporarily disabled all five configurable local MCP servers and performed a clean restart, ending the residual ChatGPT processes that remained after normal close.
    2. Codex still created app-managed Node/Node REPL helpers and rewrote config.toml, removing Node REPL’s disabled flag.
    3. Four Codex-launched Git children remained alive: rev-parse HEAD, remote -v, config --null --get core.fsmonitor, and status --porcelain.
    4. The identical direct git config --null --get core.fsmonitor command completed naturally in 91 ms.
    5. A brand-new control task was instructed to return one line and use no tools. It added two persistent Node/Node REPL helper pairs.
    6. That tool-free task left a Codex-launched git -c core.hooksPath=NUL -c core.fsmonitor=false status --porcelain process alive.
    7. The identical direct status command completed naturally in 98 ms with exit code 0.

    Earlier logs also showed automatic title/description sessions initializing complete helper/MCP environments while recording zero MCP tool executions. During the full recurrence, ten matching local helper suites remained resident simultaneously.

    Normal application closure was incomplete twice during the follow-up. Residual ChatGPT processes had to be ended through Task Manager; one idle residual group used approximately 755 MB.

    This evidence supports a compound lifecycle problem:

    • Git commands launched through Codex can remain stuck even though identical direct commands complete immediately.
    • Visible and hidden sessions can retain eager MCP/helper environments after their work finishes.
    • The retained processes increase memory and process-table pressure.
    • Repeated cleanup through taskkill /T then enumerates that enlarged process table through WMI and can escalate into the reported storm.

    After restoring the original MCP configuration and performing a final clean restart, the app was stable with one expected helper suite, zero Git, zero taskkill, and zero exact Win32_Process WMI failures during the final observation. This cleared the accumulated state but did not fix the underlying product defect.

    I can provide the full WPR ETL, targeted WPA CSV exports, event-log details, process command lines, and timestamped chronology if useful.

    Existing OpenAI feedback ID: 019f7bb3-69e7-7f23-8635-79b16e2eb930.

    Related reports: #29408, #33531, and #34251.

    Chronology note: #29408 documented the same Git polling command family on Codex versions 26.616.6631.0, 26.623.11225.0, and 26.707.62119. This indicates that the underlying Git polling defect predates 26.715; the WPR evidence above in this comment connects that older behavior to the downstream taskkill/WMI storm.

  9. athan614 commented on Jul 20, 2026

    @athan614

    Same issue here, ChatGPT/Codex windows app is stuck in a runaway process-enumeration loop with taskkill.exe whenever launched, causing WMI to get stuck at 25% CPU constantly, lagging my entire PC.

  10. q447552640 commented on Jul 21, 2026

    @q447552640

    If I launch Codex Desktop directly in user mode, it spawns a large number of terminal processes and keeps failing repeatedly, with up to five or six hundred such processes running. Meanwhile, the WMI Provider Host process consumes over 50% of system resources. However, when I run it as administrator, no excessive terminal processes are created and WMI operates normally. This issue is most likely caused by a permission-checking process that keeps failing and restarting continuously.
    Right Click Codex And Run As Administrator may be helpful.

  11. 13 remaining items

  12. dciechanowicz commented on Jul 23, 2026

    @dciechanowicz

    Independent reproduction on the same Windows desktop build confirms this issue, including the cold/idle startup variant.

    Environment

    • Codex Desktop package: OpenAI.Codex_26.715.4045.0_x64__2p2nqsd0c76g0
    • Bundled cleanup runner observed: codex-command-runner-0.145.0-alpha.18
    • Windows 11 x64
    • 32 GB RAM

    Observed behavior

    Merely launching Codex/ChatGPT Desktop, without intentionally starting a task, caused process, thread, handle, and memory counts to grow continuously. Fully closing the Codex/ChatGPT processes caused the counts to fall back toward baseline.

    A live snapshot during the storm showed:

    • Total processes: 1,669
    • Total threads: 11,179
    • Total handles: 398,810
    • taskkill.exe: 618 processes / 114,606 aggregate handles
    • conhost.exe: 649 processes / 63,534 aggregate handles

    Earlier before/after screenshots from the same launch showed growth within approximately 101 seconds from:

    • Processes: 313 -> 871
    • Threads: 5,079 -> 8,701
    • Handles: 147,879 -> 251,813
    • Used RAM: 11.4 GB -> 17.6 GB

    The counts continued increasing after that snapshot.

    Parent-process confirmation

    A WMI-independent Toolhelp32 process-tree capture was used because WMI became too slow to answer during the storm. In one diagnostic sample, all 587 live taskkill.exe instances had the Codex Desktop ChatGPT.exe process (PID 28712) as their direct parent:

    Parent: 28712, ChatGPT
    ChildCount: 587
    ChildHandles: 110604
    

    Closing Codex stopped the spawning and returned the system to approximately 380 processes / 5,000 threads / 150,000 handles.

    This independently supports the existing diagnosis that the desktop cleanup path can enter an unbounded taskkill.exe spawn loop. It also confirms that the trigger can occur immediately after restoring/launching the desktop app, before the user intentionally initiates a new task.

    A sanitized diagnostic ZIP containing six-sample process timelines, parent PIDs, grouped handle/thread/RAM counts, app versions, and no chat/config contents is available if maintainers need it.

  13. levgoriachev commented on Jul 23, 2026

    @levgoriachev

    26.715.72359 still exist. Codex dev folks, do you realize that it's a blocker not even major bug? The desktop app is complitly unuseable. Like at all =)
    And we all still paying subscription for the unuseable app. Is there any workarounds maybe? till the official fix.

  14. DaveW001 commented on Jul 23, 2026

    @DaveW001

    Independent reproduction: 26.715.10079.0, WmiPrvSE at 49% CPU, 39k CPU-seconds

    Same failure mode on a newer Store build.

    Environment

    • Codex Desktop: OpenAI.Codex_26.715.10079.0_x64__2p2nqsd0c76g0
    • Windows 11 25H2 x64, build 26200
    • 32 GB RAM
    • Native Git repos, no WSL

    Live evidence from an active storm

    WmiPrvSE.exe PID 101532 accumulated 39,325 CPU-seconds in ~2 hours -- sustained 49% CPU. Three WmiPrvSE instances were running.

    Every live taskkill.exe was a direct child of ChatGPT.exe PID 85172 (Codex Desktop main process). All ran taskkill.exe /pid <pid> /t /f. 13 instances alive at capture, spawning at 3-5/sec.

    Full process tree under PID 85172:

    • 7 ChatGPT.exe (main + 6 renderer/GPU/utility/crashpad)
    • 2 codex.exe (app-server + code-mode-host)
    • 16 node_repl.exe (orphaned from closed tabs)
    • 2 powershell.exe (Codex's Electron performance sampler running WMI queries)
    • 13+ taskkill.exe + matching conhost.exe children
    • 1 git.exe hung on hash-object in one repo

    Sampler amplifies the loop

    The two powershell.exe children are Codex's [electron-sampler]. Every 30 seconds it runs:

    Get-CimInstance Win32_Process ...
    Get-CimInstance Win32_PerfFormattedData_PerfProc_Process ...
    

    This is Codex monitoring its own child processes via WMI. When the taskkill pile-up starts, these queries slow down, hit the same WMI quota, and log [electron-sampler] Failed to collect child process snapshot. Retries add more WMI load.

    So the feedback loop has three participants:

    1. Git ops hang -> Codex spawns taskkill /T
    2. taskkill queries Win32_Process via WMI -> WMI load spikes
    3. The Electron sampler independently queries Win32_Process every 30s -> adds to WMI load
    4. WMI slows -> taskkill hangs -> more taskkill spawned -> sampler retries -> accelerates

    If the sampler used CreateToolhelp32Snapshot instead of WMI on Windows, it would remove one amplifier.

    Workaround tested
    Stop-Process -Name ChatGPT,codex -Force killed the tree. WmiPrvSE CPU dropped from 49% to near zero in 30 seconds. Relaunch worked normally.

    Per-repo .git/info/exclude additions reduced Git scanning volume, but the underlying taskkill/WMI loop is still present.

  15. msg7086 commented on Jul 23, 2026

    @msg7086

    I was able to recover my working environment, by deleting all conversations and projects from codex app, i.e. performing a factory reset.

    1. Go to ~/.codex/sessions and remove all sessions. You can backup to a different directory instead of deleting them.
    2. Renaming or deleting ~/.codex/.codex-global-state.json file.

    Then when you start codex app, you should get a clean and empty workspace, and the CPU usage should drop quickly.

    If you have unfinished sessions, use codex CLI instead, resume your conversation, ask it to dump a handover markdown file. You can then pick the handover file in codex desktop app later, after the reset.

  16. PPandYY commented on Jul 24, 2026

    @PPandYY

    I published a standalone MIT-licensed containment guard for this specific Windows failure mode:

    https://github.com/PPandYY/codex-windows-taskkill-guard
    Release: https://github.com/PPandYY/codex-windows-taskkill-guard/releases/tag/v0.1.0

    This is a temporary community mitigation, not an upstream fix.

    The guard:

    • uses NtQuerySystemInformation-based native snapshots rather than WMI;
    • acts only after two consecutive samples see at least two taskkill.exe children directly owned by a package-verified Codex ChatGPT.exe;
    • fails closed unless the package family, app\ChatGPT.exe suffix, System32\taskkill.exe path, PID/creation-time identity, and direct parentage all match;
    • enters a latched zero-helper isolation state, lowers only the exact offending Codex parent to BelowNormal, runs the guard at AboveNormal, and releases after 10 quiet seconds;
    • follows Store updates by validating package identity and the package-relative executable instead of pinning a build number.

    On one affected Windows machine, under the same 30-second workload:

    • taskkill.exe peak: 26 -> 5
    • conhost.exe peak: 39 -> 17
    • WMI whole-machine CPU share: ~14.4% -> ~1.83%
    • guard CPU share: ~1.5%

    The trade-off is explicit: suppressing cleanup helpers during a storm can leave an individual tool child running. The guard never terminates Codex or conhost.exe, and never broadens its target set.

    The public repo includes 34 Python tests plus PowerShell installation/rollback fixtures. This result supports implementing a process-wide cleanup concurrency/timeout fix upstream and replacing the Desktop WMI sampler with native process APIs.

  17. levgoriachev commented on Jul 24, 2026

    @levgoriachev

    Current build still contains the disposal-to-taskkill path

    I reproduced this again after updating to the current Microsoft Store package:

    • Codex Desktop package: OpenAI.Codex_26.721.3996.0_x64__2p2nqsd0c76g0
    • Windows x64
    • Native Windows Git workspaces

    With the normal profile, the storm still starts during cold launch. Removing all pinned tasks did not prevent it: one controlled run observed 40 unique taskkill.exe processes, about 3.8 GiB peak Desktop working set, and heavy disk activity within the first minute. Direct Git control commands against the same repositories completed normally in approximately 1.5–3.4 seconds.

    Read-only inspection of this exact installed build's app.asar shows a more specific cleanup trigger than a generally slow Git repository:

    1. The local execution host exposes child stdin as an RPC-backed WritableStream.

    2. When the last RPC stub reference is disposed while the stream is not marked closed, WritableStreamStubHook.dispose() calls writer.abort(...).

    3. The local execution host's stdin abort handler invokes child cleanup.

    4. With killProcessTree=true, the Windows cleanup path invokes:

      taskkill.exe /pid <pid> /t /f
      
    5. The taskkill execFile call still has no visible hard timeout, process-wide concurrency limit, per-target single-flight guard, backoff, or circuit breaker.

    This means normal RPC stub disposal can be interpreted as cancellation and reach process-tree termination even when the read-only Git operation itself is not slow. Large/restored workspaces and long task histories increase the number of background Git/RPC lifecycles, but they are amplifiers rather than the root defect.

    A separate clean isolated CODEX_HOME with no sessions, projects, or authentication produced no taskkill storm, but still reached roughly 2.5 GiB Desktop working set and heavy plugin-related disk I/O. That separates the general startup resource problem from this cleanup defect; deleting conversation history is not a complete fix.

    Suggested product-side fixes:

    • close child stdin explicitly on normal completion before disposing the RPC stub;
    • do not treat disposal of an unused/unclosed proxy as cancellation of an already completed child;
    • add a hard timeout, global concurrency limit, and per-target deduplication around Windows tree cleanup;
    • replace or circuit-break the WMI-based process sampler after snapshot failures.

    Acceptance should include completed read-only Git commands whose RPC stubs are disposed without launching taskkill.exe, plus a stress test proving cleanup concurrency remains bounded when many background Git operations are cancelled.

    No usernames, repository names, absolute user paths, session contents, credentials, or raw traces are included. Sanitized ETW/log excerpts can be provided if a maintainer requests a specific format.

  18. Pyseph commented on Jul 24, 2026

    @Pyseph

    Independent reproduction on 26.715.10079.0

    I independently reproduced this issue using read-only monitoring that did not stop, pause, or modify any running applications.

    Evidence collected during the active slowdown:

    • 245 new taskkill.exe processes in 8 seconds (~30.6 launches/second).
    • Every observed taskkill.exe was a direct child of the packaged Codex Desktop ChatGPT.exe.
    • A separate 10-second WMI trace recorded 576 ExecQuery starts (~57.6/second).
    • The repeatedly executed query enumerated Win32_Process, including process IDs, parent IDs, working sets, thread counts, and CPU times.
    • Native CPU sampling attributed approximately 3.32 CPU cores to WmiPrvSE.exe and 1.34 cores to the Winmgmt service host.
    • Whole-system native sampling measured substantial CPU activity even while Task Manager’s aggregate display showed approximately 0%.

    User-visible impact included browser updates occurring only every few seconds, delayed typing in text fields, delayed mouse-hover feedback, and general desktop sluggishness. Memory, disk, and GPU saturation were not observed.

    The app was intentionally left running because active background work could not be interrupted, so a restart comparison was not performed. Raw ETL traces and process exports have been retained locally but are not attached publicly because they may contain local paths, identifiers, command lines, or unrelated process information. A sanitized extract can be provided through an appropriate private channel if requested.

  19. levgoriachev commented on Jul 24, 2026

    @levgoriachev

    Follow-up: process storm contained, but a separate full-rollout scan still makes Desktop unusable

    I ran controlled local experiments against the current Store build 26.721.3996.0 using a disposable portable copy. The real sessions were never deleted or modified.

    The reproduction includes an active legacy goal thread whose JSONL rollout is about 4 GiB. Removing pinned tasks did not help.

    What the experiments separated

    1. Patching the Electron cleanup path to stop spawning taskkill.exe, and disabling the WMI child-process sampler, prevented the original unbounded taskkill process storm. Process count stayed bounded.
    2. Desktop was still unusable: after the UI resumed the large thread, total Desktop working set grew from roughly 600 MiB to 4.5 GiB and the bundled codex.exe sustained roughly 100-140 MiB/s reads until an external guard terminated only the disposable build.
    3. Direct app-server tests showed that a patched cheap resume path itself can be bounded:
      • legacy thread/read(includeTurns=true): about 60 KiB returned in under 1 second;
      • thread/resume with excludeTurns=true, a 5-turn initial page, and deferred goal continuation: 61 KiB response, 6.4 seconds, about 279 MiB peak backend working set, and no automatic turn.
    4. In the full Desktop flow, immediately after thread/resume, the renderer sends thread/goal/get and then thread/goal/set. That second request starts the destructive scan.

    Specific backend path

    In app-server/src/request_processors/thread_goal_processor.rs, both goal set and clear call reconcile_thread_goal_rollout(...). It unconditionally calls reconcile_rollout(...) with no builder and no incremental items.

    In rollout/src/state_db.rs, that shape falls through to metadata::extract_metadata_from_rollout(...), which scans the rollout file. For an already materialized thread whose metadata row already exists in SQLite, this is redundant, but it rereads the complete multi-gigabyte legacy JSONL.

    The app-server trace showed thread/goal/set entering codex_rollout::recorder / rollout reconciliation at exactly the point where backend disk reads and RAM growth began.

    Suggested product-side change

    Before reconciling a goal mutation, check whether the thread metadata row already exists in StateRuntime:

    • existing metadata row: perform the goal mutation directly; SQLite is already available and the goal service reads/writes it;
    • missing metadata row: use the current rollout reconciliation as the slow migration/read-repair path, then fail if the row still does not exist.

    An acceptance test should use a large legacy rollout with existing SQLite metadata and verify that thread/resume followed by thread/goal/set reads only a bounded suffix / incremental data, does not materialize the full JSONL, and does not start a turn when goal continuation is deferred.

    This appears to be a compound failure:

    • the taskkill/WMI cleanup loop is one root defect;
    • unconditional full-rollout reconciliation on goal mutation is another;
    • a large legacy goal thread is a reliable trigger, not the underlying cause.

    The local patches were diagnostic only and are not presented as a production-ready fix. I am stopping the experiments and cleaning up the disposable build, but this should provide a narrower upstream target without requiring users to delete their conversations.

  20. ayanamislover commented on Jul 26, 2026

    @ayanamislover

    Current-build follow-up: 26.721.4979.0 isolates two repeated WMI snapshot callers

    I reproduced the remaining WMI polling on the current Windows Store package:

    • Windows 10 Pro 22H2 x64, build 19045.6466
    • Store package OpenAI.Codex_26.721.4979.0_x64__2p2nqsd0c76g0
    • The signed MSIX and app.asar were not modified

    A 10.097-second Microsoft-Windows-Kernel-Process ETW baseline recorded 14 direct powershell.exe children of the Electron main process (1.387/s), with the active WmiPrvSE.exe processes consuming about 0.286 logical cores in aggregate.

    Windows PowerShell Event 400/403 recorded only these two commands in that window:

    Get-CimInstance Win32_Process |
      Select-Object ProcessId,ParentProcessId
    
    Get-CimInstance Win32_PerfFormattedData_PerfProc_Process
    Get-CimInstance Win32_Process -Filter "ProcessId = ..."
    

    A short, reversible ChildProcess.spawn trace in the Electron main process identified the exact remaining stack:

    listProcessSubtrees (IW)
      -> RW
         -> VW  # full Win32_Process parent/child snapshot
         -> WW  # PerfProc + filtered Win32_Process detail snapshot
    

    Current bundle inspection shows two independent callers:

    1. ElectronSampler.addChildProcessFields() creates the dedicated child-process-snapshot-worker.js worker.
    2. The exported process-manager class (tX in this build) calls listProcessSubtrees from upsert/complete, retries at [0, 100, 300, 600] ms, and then falls back to a detached-process candidate scan.

    To isolate causality without restarting Codex, I temporarily neutralized only those two seams in the same Electron PID:

    • the exact worker name/file returned an empty telemetry snapshot;
    • only the process manager's listProcessSubtrees and listDetachedProcessCandidates dependencies returned empty arrays.

    The real execution host, ordinary worker construction, Git, PowerShell, and the taskkill.exe cleanup function were not globally intercepted.

    Results:

    State Window Direct PowerShell children Rate Aggregate WmiPrvSE core equivalent
    Baseline 10.097 s 14 1.387/s 0.286
    Both snapshot seams neutralized 10.063 s 0 0/s 0.000
    Restore/reapply verification 10.048 s 0 across all 9 current ChatGPT PIDs 0/s 0.023

    The last small provider-host delta was not accompanied by any child process start from any current ChatGPT PID. The Electron main PID and start time did not change, the temporary loopback Inspector was closed, and no ETW session remained.

    This is a causal isolation, not a suitable product workaround: returning empty process snapshots can make background/detached-process attribution incomplete.

    Suggested product fix:

    • replace both PowerShell/CIM paths with one coalesced native Windows snapshot (NtQuerySystemInformation or Toolhelp32 plus process metrics);
    • share/cache that snapshot between the Electron sampler and process manager instead of launching separate full-system queries;
    • pass the execution host's known OS PID into the process manager so short commands do not trigger four full retries;
    • single-flight and rate-limit fallback discovery, and stop retrying once the command has already completed without a usable PID;
    • keep cleanup bounded independently so a failed snapshot cannot amplify into a taskkill.exe/WMI feedback loop.

    The Electron Desktop implementation is not present in the public repository, so I cannot submit the product-code PR against openai/codex. This comment provides the current-build call sites and controlled A/B for the private Desktop implementation.

  21. chess-oai commented on Aug 5, 2026

    @chess-oai
    Contributor

    while we believe 26.730.7989.0 substantially improves the WMI storm, there are still many cases of taskkill floods that we need to resolve separately. leaving this issue open to track taskkill issues, but if you were suffering mostly WMI-related system exhaustion, re-test on 26.730.7989.0

  22. DukeTwoCan commented on Aug 13, 2026

    @DukeTwoCan

    I can confirm that I am still experiencing this issue on Windows 11 Pro, and my codex version is 26.803.10989.0. As soon as I launch the ChatGPT/Codex desktop application, it spawns a bunch of taskkill/git tasks and then my WMI Provider Host process spikes. If left alone it causes a system crash, but if I kill the ChatGPT/Codex desktop application in task manager the WMI storm subsides and everything goes back to before I tried to launch.

  23. DukeTwoCan commented on Aug 14, 2026

    @DukeTwoCan

    Windows 11 Pro, updated to ChatGPT/Codex desktop version 26.810.41047 and launched without seeing these issues anymore.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Papercuts 2026appIssues related to the Codex desktop appbugSomething isn't workingperformancetool-callsIssues related to tool callingwindows-osIssues related to Codex on Windows systems

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions