Skip to content

VS Code Codex extension repeatedly broadcasts huge thread-stream-state-changed snapshots for completed long threads, causing extension-host/renderer CPU spikes #20781

Open

Description

@duncanmcl

What version of the IDE extension are you using?

26.429.30905

What subscription do you have?

Pro

Which IDE are you using?

VS Code

What platform is your computer?

Linux 6.8.0-110-generic x86_64 x86_64

What issue are you seeing?

The VS Code Codex extension can enter a high-CPU state when a long completed thread is opened or resumed. The extension repeatedly broadcasts thread-stream-state-changed messages with:

change.type = snapshot
params.change.conversationState = full conversation state

For one long local thread, each snapshot frame was about 116 MB. Repeated snapshots drove the VS Code NodeService / extension-host process to 250-400% CPU while idle or while switching/resuming threads.

This does not look like workspace indexing, project scanning, or app-server compute. The hot stack is Node stream/Buffer copying in the VS Code extension host while receiving/sending huge IPC frames.

Representative process sample before mitigation:

PID 3261642 code NodeService: 340-405% CPU
PID 3261826 codex app-server: 0-1% CPU

Representative Codex log line for the long thread:

maybe_resume_success
conversationId=019dd343-efa8-7093-8746-772db1e1ade7
latestTurnStatus=completed
markedStreaming=true
turnCount=1294+

IPC capture showed repeated huge frames:

method: thread-stream-state-changed
change.type: snapshot
frameBytes: about 116,220,000 to 116,550,000
large field: params.change.conversationState
conversationState bytes: about 116 MB

Normal healthy patch updates were tiny:

method: thread-stream-state-changed
change.type: patches
frameBytes: hundreds of bytes to a few KB

Largest observed frame shape, with conversation text omitted/redacted:

{
  "method": "thread-stream-state-changed",
  "changeType": "snapshot",
  "frameBytes": 116249683,
  "paramsBytes": 116249542,
  "changeBytes": 116249409,
  "fieldBytes": {
    "type": 10,
    "conversationState": 116249369
  }
}

CPU profile hot stack:

onStreamRead
Readable.push
addChunk
Buffer.concat
_copyActual
createUnsafeBuffer

Native perf sample showed:

node::StreamBase::CallJSOnreadMethod
node::EmitToJSStreamListener::OnStreamRead
node::LibuvStreamWrap::OnUvRead
__memmove_avx_unaligned_erms

strace -f -c against the hot VS Code NodeService showed very little file I/O:

read calls:   222
write calls:  175
dominant time: futex/epoll/memory activity

That argues against filesystem scanning as the cause.

After a local mitigation and VS Code reload, the extension log showed the large snapshot was being skipped:

Skipping large conversation snapshot broadcast
conversationId=019dd343-efa8-7093-8746-772db1e1ade7
turnCount=1307

After that, idle CPU dropped back to low/zero. CPU can still rise while the webview actively renders streamed answer text, but the idle full-snapshot storm stopped.

What steps can reproduce the bug?

The most reliable local trigger was:

  1. Have a long local Codex thread with about 1300 turns.
  2. Restart or reload VS Code.
  3. Open/resume the long thread in the Codex VS Code extension.
  4. Watch window1/exthost/openai.chatgpt/Codex.log.
  5. Observe a line like:
maybe_resume_success ... latestTurnStatus=completed markedStreaming=true turnCount=...
  1. Observe repeated thread-stream-state-changed warnings and high CPU.
  2. Instrument IPC or profile CPU to see huge snapshot frames and Buffer.concat hot stacks.

The relevant installed webview bundle code appears to be this reconnect/status path:

this.ipcBridge.registerBroadcastHandler(`client-status-changed`, e => {
  if (e.params.status === `connected`) {
    for (let e of this.streamingConversations) this.broadcastConversationSnapshot(e);
    return
  }
  ...
})

The resume path marks the conversation as streaming and broadcasts a snapshot even when the latest turn is completed:

e.markConversationStreaming(t),
e.setConversationStreamRole(t,{role:`owner`}),
e.broadcastConversationSnapshot(t);
V.info(`maybe_resume_success`, {
  safe: {
    conversationId:t,
    turnCount:ee.length,
    latestTurnStatus:F?.status??null,
    markedStreaming:true
  }
})

The full snapshot path sends the whole conversation:

broadcastConversationSnapshot(e){
  if(this.getStreamRole(e)?.role!==`owner`)return;
  let t=this.conversations.get(e);
  t && this.dispatchMessageFromView(
    `thread-stream-state-changed`,
    {
      conversationId:e,
      hostId:this.hostId,
      change:{type:`snapshot`,conversationState:t},
      version:ae(`thread-stream-state-changed`)
    }
  )
}

The patch path is already efficient and small:

broadcastIpcStatePatches(e,t){
  t.length!==0 &&
  this.getStreamRole(e)?.role===`owner` &&
  this.dispatchMessageFromView(
    `thread-stream-state-changed`,
    {
      conversationId:e,
      hostId:this.hostId,
      change:{type:`patches`,patches:t},
      version:ae(`thread-stream-state-changed`)
    }
  )
}

Likely root cause:

completed long thread remains in streamingConversations
-> client-status-changed/resume broadcasts full snapshot
-> full conversationState is about 116 MB
-> extension host spends CPU copying/parsing/rendering huge IPC frames

What is the expected behavior?

The extension should not repeatedly broadcast the entire conversation state for long completed threads.

Expected behavior:

completed resumed thread should not remain in streamingConversations
reconnect/client-status events should not send full snapshots for huge threads
long threads should use bounded/delta/paginated hydration
idle CPU should return to near zero after resume/render is complete

Suggested upstream fix:

  1. Do not call markConversationStreaming for resumed threads whose latest turn is completed.
  2. Do not include completed long threads in streamingConversations.
  3. On client-status-changed: connected, do not rebroadcast full snapshots for completed/non-active threads.
  4. Replace full-snapshot broadcast for long threads with bounded/paginated hydration or patch-only sync.
  5. Add a defensive max snapshot size before IPC dispatch.
  6. Ensure thread-stream-state-changed warnings do not flood logs when no handler is present.

An upstream guard could be shaped like:

if conversation is completed and not actively generating:
  do not mark streaming
  do not broadcast snapshot on reconnect

if snapshot would exceed a bounded IPC size:
  use paginated state sync or request/response hydration instead of broadcast

Additional information

I applied a local mitigation to confirm the diagnosis. This is not the ideal upstream fix, but it stopped the local CPU storm.

Patched local file:

/home/dun/.vscode/extensions/openai.chatgpt-26.429.30905-linux-x64/webview/assets/app-server-manager-signals-D5l5fvZj.js

Local mitigation:

broadcastConversationSnapshot(e){
  if(this.getStreamRole(e)?.role!==`owner`)return;
  let t=this.conversations.get(e);
  if(!t)return;

  // Cheap guard first. Do not JSON.stringify a huge conversation just to decide
  // whether to skip it.
  let n=t?.turns?.length??0;
  if(n>100){
    V.warning(`Skipping large conversation snapshot broadcast`,{
      safe:{conversationId:e,turnCount:n},
      sensitive:{}
    });
    return
  }

  // Backstop for small/medium threads with unexpectedly huge embedded state.
  let r=0;
  try{r=JSON.stringify(t).length}catch{}
  if(r>5e6){
    V.warning(`Skipping oversized conversation snapshot broadcast`,{
      safe:{conversationId:e,snapshotBytes:r,turnCount:n},
      sensitive:{}
    });
    return
  }

  this.dispatchMessageFromView(`thread-stream-state-changed`,{
    conversationId:e,
    hostId:this.hostId,
    change:{type:`snapshot`,conversationState:t},
    version:ae(`thread-stream-state-changed`)
  })
}

Important detail: warning suppression alone is not enough. Some existing workarounds suppress:

[IpcClient] Received broadcast but no handler is configured method=thread-stream-state-changed

That reduces log spam, but it does not stop the underlying huge conversationState broadcast. In this case the dominant cost was the 116 MB IPC frame and Buffer/JSON processing, not just the warning log line.

Activity

  1. added
    bugSomething isn't working
    app-serverIssues involving app server protocol or interfaces
    on May 2, 2026
  2. rdylina commented on May 10, 2026

    @rdylina

    Seeing the same mechanism in Codex Desktop on macOS, not just the VS Code extension: thread-stream-state-changed tried to send full conversation state, payloadBytes=405234701, then dropped it because maxIpcFrameBytes=268435456. The app-server built/resumed the huge state first, and IPC/renderer became the failure surface, so this likely needs app-server-side bounded snapshots in addition to client rendering fixes.

  3. rdylina commented on May 10, 2026

    @rdylina

    Follow-up / PR offer: I read the contribution docs and understand external PRs are invitation-only. If the team thinks this direction fits, I’d be happy to take a stab at an invited PR for the open-source app-server side.

    Conceptually, the fix I’d propose is:

    • in no case should opening a thread load the entire transcript into app-server + client state
    • open threads from metadata + the latest tail only, roughly the most recent 100-200 messages or a small byte cap, whichever comes first
    • lazy-load older chunks only when the user scrolls upward / explicitly asks for them
    • compact or summarize the old head for model context instead of treating the full durable transcript as active state
    • keep large tool outputs/artifacts behind references, not inline in normal thread state
    • add hard byte/count limits so thread/read, thread/resume, and stream-state snapshots cannot produce 100MB+ payloads
    • document that Desktop/IDE clients should use paged turns/items and never full-hydrate a completed thread just because it was clicked

    The desktop side still needs closed-code changes, but the app-server can provide the safe bounded contract and fail gracefully when a client asks for full hydration of an oversized rollout.

  4. HyperSpAceBoy commented on Jul 13, 2026

    @HyperSpAceBoy

    Still reproducible in current Windows Desktop build (2026-07-13)

    This same full-state snapshot mechanism remains present in the Microsoft Store Windows package 26.707.8479.0 with bundled app-server 0.144.2.

    A deterministic Desktop reproduction is now available in #32722:

    1. Keep several completed/large conversations resident.
    2. Open an otherwise empty new Codex Desktop window.
    3. The new client connection causes the packaged client to iterate every streamingConversations entry and call broadcastConversationSnapshot.
    4. Each broadcast still contains the complete in-memory conversationState.
    5. The 256 MiB IPC check is applied only after JSON.stringify, so the expensive full-state construction and serialization have already occurred.

    Measured during this trigger:

    • app-server: 624.4% CPU
    • app-server working set: 30.7 GB
    • Git starts: 0
    • user-visible impact: multi-second whole-desktop pointer/UI stalls even though the newly opened task is empty

    This is not a low-spec-machine or repository-size explanation. The empty window is not the payload; unrelated resident histories are fanned out to it.

    The July Desktop changelog does not name this path as fixed, and the current production bundle confirms it is not fixed. Please prioritize a bounded protocol: selected-thread delivery, metadata/tail-first hydration, byte/count limits before serialization, and patch/delta updates rather than full snapshots on every client connection.

  5. TheTurboForge commented on Jul 27, 2026

    @TheTurboForge

    Additional client-routing evidence from extension 26.707.41301

    I reproduced the thread-stream-state-changed warning path on Linux with:

    • VS Code 1.128.0
    • openai.chatgpt 26.707.41301
    • extension-bundled app-server initialized normally

    A trace of the shipped extension bundle identified two related client-routing problems:

    1. The extension creates one global/service IPC client in addition to the real
      webview clients. Stream-state patches are broadcast to every other client,
      including the service client, even though that client has no handler for
      thread-stream-state-changed. This produces one false no-handler warning
      per delivered patch.

    2. When a client connects, reconnect hydration iterates the retained streaming
      conversations and broadcasts full snapshots rather than targeting the newly
      connected webview. Existing clients and the service client therefore receive
      hydration that was intended only for the newcomer.

    I tested a reversible local mitigation that:

    • assigns the global IPC singleton a service-only client type;
    • excludes service clients from stream-state delivery;
    • ignores self/service connections as reconnect-hydration triggers;
    • sends each reconnect snapshot only to the newly connected webview.

    After a controlled restart, conversation resume and the next streamed turn both
    worked. The fresh post-change log contained zero
    thread-stream-state-changed no-handler warnings, compared with 1,799 in the
    accumulated pre-change log.

    Important boundaries:

    • I did not measure snapshot frame sizes in this test.
    • This does not prove a CPU or renderer-crash fix.
    • ResizeObserver errors continued after the no-handler warnings disappeared.
    • Missing-turn errors were also absent afterward, but I cannot attribute that
      result solely to this mitigation.

    The upstream contract I would suggest is:

    • give IPC clients an explicit role/capability set;
    • deliver broadcasts only to clients that implement the message contract;
    • make reconnect hydration a targeted response to the connecting webview;
    • separately bound or page conversation snapshots before serialization.

    This appears complementary to the byte/count limits already proposed here:
    bounded payloads prevent oversized transfers, while targeted delivery prevents
    valid payloads from being fanned out to unrelated clients.

  6. LukeMainwaring0 commented on Aug 5, 2026

    @LukeMainwaring0

    Current macOS Desktop evidence: the oversized snapshot path now reaches a V8 OOM crash

    I reproduced this same underlying mechanism in the current macOS Codex desktop build:

    • Codex/ChatGPT desktop: 26.730.61639 (6234)
    • Codex Framework / Chromium: 151.0.7922.71
    • Apple silicon
    • Crash time: 2026-08-05 11:35:21 PDT
    • Crash incident: AE85AAEF-58AE-433A-82FE-409D59CE6945

    For the final 64 seconds before the crash, the desktop log emitted 15 warnings of this form:

    Dropping broadcast: payload too large
    maxIpcFrameBytes=268435456
    method=thread-stream-state-changed
    

    The rejected payload grew monotonically from 269,908,281 bytes to 296,373,565 bytes, with the broadcast rebuilt approximately every 4-6 seconds.

    The process then crashed with EXC_BREAKPOINT (SIGTRAP) on CrBrowserMain. Disassembly of the crashing Codex Framework frame at offset 0x4469b18 identified the Node/Electron OOM handler in:

    ../../owl/common/node_bindings.cc
    OOM detail:
    OOM error in V8:
    

    The handler terminates via brk #0. Nearby V8 strings include:

    Allocation failed - process out of memory
    Allocation failed - JavaScript heap out of memory
    

    macOS did not jetsam or kill the process. A resource diagnostic showed a maximum bundled Codex footprint of approximately 8.58 GB. The app restarted automatically and the active task recovered.

    The first over-limit warning immediately followed a reasoning-state update from one active long-running task, while multiple active or completed histories were simultaneously represented as streaming. That task was the trigger, not the root fault. The observed failure chain was:

    retained streaming histories
    -> aggregate thread-stream-state-changed payload constructed/serialized
    -> payload rejected only after exceeding 256 MiB
    -> same large temporary allocation repeated every few seconds
    -> V8 OOM handler terminates the desktop app
    

    This upgrades the demonstrated impact from CPU/memory churn to a full application crash in build 26.730.61639. The corrective boundary still appears to be:

    • clear stale streaming state for completed/disconnected tasks;
    • send deltas or targeted hydration instead of aggregate snapshots;
    • page/cap historical state;
    • apply byte/count limits before allocation/serialization;
    • add backpressure so an over-limit update is not rebuilt continuously.

    I also submitted the redacted evidence to OpenAI Support. The raw desktop log and full .ips crash report are preserved and can be provided privately to a maintainer; they are not attached here because the raw log contains unrelated task metadata.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    app-serverIssues involving app server protocol or interfacesbugSomething isn't workingextensionIssues related to the VS Code extensionperformance

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions