T3 Code Nightly reaches approximately 11.5 GB while concurrent agent progress stalls

What happened

I had two threads running in T3 Code Nightly with several sub-agents, estimated at roughly four, though the exact number is uncertain. Task Manager showed T3 Code (Nightly).exe using approximately 11,557 MB.

The app remained usable, but the agents and sub-agent threads stopped responding or making visible progress. Restarting T3 Code helped temporarily, then the lag returned.

The incident was reported as approximately nine hours before triage began on October 11, 2026 at 22:21 UTC, placing the estimated onset around 13:20 UTC. This time is approximate; the screenshot is not timestamped.

Diagnosis

The running desktop backend identified itself as 0.0.46-nightly.20261011.2967. The separate npx t3 triage CLI reported 0.0.45; that CLI version should not be mistaken for the affected desktop version.

Read-only inspection covered these two T3 threads:

9e2fb61f-d607-4a00-951a-bb17a84721b9
1a15d915-01c0-463f-952e-d61b2ff1094e

The provider capture associated with the first thread contains substantial activity between 12:20 and 14:20 UTC on October 11: 8,346 item-start notifications, 8,337 item-completion notifications, 3,787 token-usage updates, and 175 collaboration-agent tool-call items. Structured records reference 104 distinct agentThreadId values. This capture includes many internal Codex thread IDs; these are aggregate events over the window, not a count of simultaneously running sub-agents or proof that all events belong to the first T3 thread.

Within that window, the capture contains 33 retryable response-stream disconnect errors, one non-retryable account usage-limit error at 13:44:26.756 UTC, and a non-retryable retry-limit error with HTTP 429 at 13:44:38.658 UTC. These failures can explain interrupted provider progress, but do not establish why the T3 process group reached 11.5 GB.

The second thread's provider capture has no entries between 12:20 and 14:20 UTC, so it cannot corroborate activity at the estimated onset. It contains two later retryable response-stream disconnects.

Renderer history samples stayed below approximately 750 MiB; the investigation could not identify which process accounted for the remaining memory. Long-lived subscription durations were excluded as evidence of latency because they can be normal connection lifetimes.

The memory-growth cause remains unproven. High event volume, provider errors, and concurrent agent activity are diagnostic leads, not a confirmed memory leak or root cause. Exact parent-child relationships and concurrent child counts were not established.

Steps to reproduce

Reported workload; a minimal deterministic reproduction has not been established:

Launch T3 Code Nightly on Windows.
Run two agent threads concurrently with several sub-agents.
Allow the workloads to continue and monitor Task Manager and agent progress.
Observe whether T3 Code memory grows substantially while agent progress stalls, even though the app remains usable.
Restart T3 Code and observe whether progress improves temporarily before the lag returns.

Expected: agent progress remains responsive during concurrent workloads, with bounded memory usage.

Actual: T3 Code reached approximately 11,557 MB; agent progress stopped responding, and restarting the app provided only temporary relief.

Version

Affected desktop backend: 0.0.46-nightly.20261011.2967

Release commit: 8d199b4f7b918079161d5f43e0146c099301be08

Triage CLI: 0.0.45

Environment

Windows x64, OS version 10.0.26200.
Node v26.8.2.
T3 Code Nightly desktop app with a local backend.
Codex provider activity was present in both inspected captures.

Evidence

Task Manager screenshot: T3 Code (Nightly).exe at approximately 11,557 MB and 0.9% CPU.

Server trace: resourceAttributes.service.version = 0.0.46-nightly.20261011.2967.

First thread provider capture, filtered to 12:20–14:20 UTC: 8,346 item starts; 8,337 item completions; 3,787 token-usage updates; 175 collaboration-agent tool-call items; 104 unique structured agentThreadId values; 33 retryable responseStreamDisconnected errors; usage-limit error at 13:44:26.756 UTC; responseTooManyFailedAttempts with HTTP 429 at 13:44:38.658 UTC.

Second thread provider capture: no entries from 12:20–14:20 UTC; later responseStreamDisconnected events were retryable.

Renderer history: maximum observed sampled working set approximately 691 MiB; maximum observed peak approximately 748 MiB.

Provider counters cover a shared capture and should not be interpreted as per-thread concurrency. Full provider logs are not attached because they may contain private prompts and other sensitive data. The screenshot should be attached manually.

Related issues

#5681: related report of concurrent Codex sub-agent progress overwhelming event ingestion. Possible connection, not a confirmed duplicate. The proposed mitigation in PR #5682 was closed without merging.
#7722: older Windows report of stuck Codex sub-agent progress; similar symptoms, but a different version and no confirmed matching memory-growth mechanism.

Fix applied or workaround

The user restarted T3 Code before triage, which helped briefly before the lag returned. No update, configuration change, source patch, database write, or diagnostic restart was performed during triage.

Filed by

Draft prepared by Codex (GPT-6.1 Sol), via t3 triage.

Note

Grok responding on behalf of Julius.

There isn't enough here to act on yet, so I'm marking this as needing more info. The stall lines up with what your capture shows: a Codex account usage-limit error at 13:44:26 UTC, then responseTooManyFailedAttempts (HTTP 429) 12 seconds later. That comes from your account or upstream, not from T3. Your renderer stayed under ~750 MiB. Task Manager's "T3 Code (Nightly).exe" group also counts the child processes, including the Codex app-server and any MCP servers it starts, so the 11.5 GB hasn't been tied to T3 itself.

Could you answer these?

In Task Manager's Details tab, which process names and PIDs hold the memory (codex, node, the Electron main process, or something else)?
Does memory keep growing after usage limits reset and progress resumes, or only while requests are failing?
Does it happen with a single thread and no sub-agents?

Related: #17295 (renderer growth on macOS) and #15979 (Codex MCP servers that keep running).

Thanks for the questions:

The screenshot is from Task Manager's Processes tab, where the app shows up as a single grouped entry, T3 Code (Nightly).exe, at about 11,557 MB memory and 0.9% CPU (0 MB/s disk, 0 Mbps network). The group was collapsed, so I can't tell you which child process (Electron main, node, codex, etc.) or PID holds the memory. I haven't checked the Details tab.
Yes, it still happens after the usage limits reset and progress resumes, so it isn't only while requests are failing.
No, I haven't seen it with a single thread and no sub-agents.

@Sovex666

A second Windows data point for question 1 ("which process holds the memory"), plus a way to get that breakdown without expanding Task Manager by hand.

On this machine the desktop Nightly (product version 0.0.46, I did not read the exact nightly build) has been up 17.9 hours. Windows 11 Pro, 126 GB RAM, state database 1.24 GB, one turn running and one queued at the moment of the sample, no stall. Working set of the whole process tree under T3 Code (Nightly).exe: 5,433 MB in 35 processes.

Process kind	Count	Sum	Largest
Electron renderer	1	3,069 MB	3,069 MB
`claude.exe` (provider)	2	651 MB	350 MB
Electron without a `--type` flag (main, and I assume the bundled server)	2	613 MB	386 MB
`python.exe` + `uv.exe` (probably MCP servers; I did not check their command lines)	10	318 MB	94 MB
Electron GPU process	1	184 MB	184 MB
Electron utility	2	179 MB	107 MB
`chrome-headless-shell.exe`	3	167 MB	73 MB
everything else (shells, conhost, cloudflared, resource monitor)	14	253 MB	85 MB

So here the renderer is more than half of the tree, about four times the ~750 MiB reported above, while the provider children are small. That is one sample with a different workload (one running turn), not a reproduction of the 11.5 GB case, and I have no earlier sample to say whether the renderer grew over the 17.9 hours.

@Sovex666 if it helps, this is the read-only PowerShell that produced the table. It prints one line per process kind for the tree under the app, so the next time the group is large you can see in a few seconds whether it is the renderer, the server, or codex:

PowerShell

$all = Get-CimInstance Win32_Process | Select-Object ProcessId, ParentProcessId, Name, WorkingSetSize, CreationDate, CommandLine
$byParent = $all | Group-Object ParentProcessId -AsHashTable
function Tree($id) { $out = @(); foreach ($c in @($byParent[[uint32]$id])) { if ($c) { $out += $c; $out += Tree $c.ProcessId } }; $out }
$roots = @()
$roots += $all | Where-Object { $_.Name -like 'T3 Code*' -and -not ($all.ProcessId -contains $_.ParentProcessId -and ($all | Where-Object ProcessId -eq $_.ParentProcessId).Name -like 'T3 Code*') }
foreach ($r in $roots) {
  $t = @($r) + (Tree $r.ProcessId)
  $age = [math]::Round(((Get-Date) - $r.CreationDate).TotalHours, 1)
  "{0} pid-root age {1} h: {2} processes, {3:N0} MB total" -f $r.Name, $age, $t.Count, (($t | Measure-Object WorkingSetSize -Sum).Sum / 1MB)
  $t | ForEach-Object {
    $kind = $_.Name
    if ($_.Name -like 'T3 Code*') { if ($_.CommandLine -match '--type=([a-z-]+)') { $kind = "electron " + $Matches[1] } else { $kind = 'electron main' } }
    elseif ($_.Name -eq 'node.exe' -and $_.CommandLine -match 'mcp') { $kind = 'node (mcp)' }
    [pscustomobject]@{ Kind = $kind; WS = $_.WorkingSetSize }
  } | Group-Object Kind | ForEach-Object { "   {0,-28} n={1,-3} sum {2,7:N0} MB  max {3,6:N0} MB" -f $_.Name, $_.Count, (($_.Group | Measure-Object WS -Sum).Sum / 1MB), (($_.Group | Measure-Object WS -Maximum).Maximum / 1MB) } | Sort-Object
}

Not checked: which thread or view the renderer memory belongs to, and whether it falls after a window reload.

Posted by Claude (Opus 5.5, Claude Code via T3 Code) on behalf of @AdEx-Partners-DE.

Following up with the process breakdown you requested. During another high-memory episode on the same Windows machine, inspection showed that most memory was held by MCP helpers beneath T3’s shared Codex app-server, rather than the Electron renderer.

Under codex.exe PID 10480, I measured:

138 Node processes: ~11.3 GiB private memory.
68 Python processes: ~2.0 GiB private memory.
34 uv processes: ~2.9 GiB private memory.

Including smaller helpers, that subtree totaled approximately 16.3 GiB private memory. Its summed working sets were approximately 9.4 GiB, although that can double-count shared pages. Private memory is committed allocation, not the same metric as Task Manager’s Processes-tab memory column.

There were 17–18 instances of several MCP servers, including GitNexus, Context7, Playwright, and filesystem. The helpers remained descendants of the live Codex app-server; they were not orphaned processes.

I understand from #21905 that completed sessions can retain MCP processes for approximately 60 minutes under the defaults. These measurements establish where the memory is going, but do not yet distinguish expected retention from delayed cleanup beyond that window.

With this many configured MCP servers, retained session trees create substantial memory pressure during multi-agent use.

This is a new live measurement, not retrospective proof of which processes held memory during the original October 11 incident.

@juliusmarminge

@juliusmarminge Another report now windows was only used to see the work on the mac

Windows desktop: slow thread opening and scrolling/navigation stalls

Issues and where they appear

In T3 Code Nightly 0.1.4-nightly.20261027.3783 on Windows, a large thread hosted by a separate Mac environment has two visible problems:

Opening the thread: the reporter estimates roughly three seconds between selecting it and seeing its conversation content. A Performance trace captured a 1,961.796 ms interaction-to-next-paint for the selection click, including a 1,682.603 ms renderer-main task. Click-to-content-visible time has not been measured.
Reading and navigating the thread: scrolling upward through the center conversation timeline and moving around the UI become sluggish while the agent is active. Four later renderer-main tasks in the scrolling scenario lasted 1,061.3, 898.3, 723.0 and 891.1 ms. Concurrent updates prevent attributing each task specifically to wheel input.

The trace identifies substantial synchronous work in the Windows renderer during both scenarios. Work-entry failure classification dominated the sampled execution. The expensive entry, input, branch and invocation pattern remain unknown.

This was observed on one Windows machine. Other client platforms were not profiled, so Windows exclusivity is not established.

Environment

Windows 11 Pro, 64-bit, build 26200; 16 logical processors; approximately 32 GiB RAM.
Installed desktop client: Nightly 0.1.4-nightly.20261027.3783, verified through executable and packaged root metadata.
Agent execution: a separate Mac environment, according to the reporter; the desktop target was identified as Mac-hosted. The Mac server version and exact connection route are unknown.
Source references below identify the corresponding code paths in the repository. The installed client's exact source commit is unknown.

Reproduction scenario

Run this reproduction on a physical Windows machine with a screen attached, using the Windows desktop client interactively. A Windows VM, Linux VM, headless environment, or Mac client is not an acceptable substitute for this validation. The Mac may still host the remote agent.

Connect the Windows Nightly desktop client to the Mac environment.
Select an existing large thread with extensive tool activity while a generation is active. Measure selection-to-first-visible-conversation separately from interaction-to-next-paint, and note whether the selection indicator or loading state responds immediately.
Scroll upward through the center conversation timeline to read earlier content while updates continue.
Observe delays in scrolling and other UI interactions.

The affected thread had a running generation and a large visible history containing aggregated tool activity. The later workload count below measures persisted records; client-loaded counts, DOM counts, wire payload sizes and update rates were not measured. This is an observed scenario, not a deterministic minimal reproduction. Whether the opening delay occurs on every switch, only on first entry, after a cache miss or after reconnecting is unknown.

Expected behavior: opening and navigating a long remote thread should keep the client responsive, including while updates arrive.

Affected thread workload

Inspection of the affected thread at 2026-10-28 17:29:34 UTC recorded the following workload:

323 messages: 134 user and 189 assistant messages.
7,518 tool-call execution nodes.
45 subagent records: 39 app-owned and 6 provider-native. Their recorded statuses were 36 completed, 3 failed, 3 interrupted and 3 running.
107 runs and 8,271 timeline items, including 3,793 command-execution items, 3,614 dynamic-tool items and 112 file-change items.
270,005,726 bytes (about 257.5 MiB) of stored timeline-item JSON. The largest timeline-item JSON record was 1,050,263 bytes; the largest command-output string was 1,043,396 characters.

Tool-call counts use unique execution nodes with kind='tool_call'; messages use message records grouped by role; subagents use distinct subagent records. Timeline items are a separate representation and are not added to the tool-call count. Stored JSON sizes do not measure transferred payloads, renderer memory or mounted UI rows. The workload was still changing, so these counts are a point-in-time snapshot rather than a fixed reproduction input.

All 45 recorded child-thread references resolved to thread rows. They represent 40 unique descendant thread IDs because multiple subagent records can refer to the same child thread.

Captured renderer path

The affected selection click produced these Performance timings:

Interaction-to-next-paint: 1,961.796 ms.
Click processing: 1,680.445 ms.
Enclosing renderer-main task: 1,682.603 ms.
V8 sample weight attributed to failure classification within that task: approximately 1,435 ms.

Sample weights are statistical estimates, not exact per-call timings. Interaction-to-next-paint does not identify the first frame containing conversation content.

The sampled work corresponds to workEntryIndicatesToolFailureFromOutput at packages/client-runtime/src/work-log/presentation.ts:258.

The caller corresponds to workEntryIsVisibleInGroup at apps/web/src/components/chat/MessagesTimeline.logic.ts:267-278. This places the expensive classification work in timeline visibility/status derivation on the UI thread.

During the later scrolling scenario, the same classifier was the dominant sampled leaf across the interval containing the four long tasks listed above. Those tasks were dominated by microtasks. The trace does not distinguish a single expensive input or regex branch from repeated classification calls, or a combination of both.

Supporting observations and measurement limits

An earlier process sample recorded 15.922 renderer CPU-seconds over 15.669 wall-seconds during two scroll inputs, snapshots and ongoing updates. This is approximately one logical CPU of activity, or 6.35% of nominal capacity on this 16-logical-processor machine. It is a process-wide measurement and does not independently establish main-thread saturation. The renderer was the largest measured CPU consumer among the sampled T3 processes.

A shorter, naturally quiet Mac-hosted thread recorded 188.768 ms interaction-to-next-paint for one selection click. Both content and activity differed, so this is not a controlled comparison. Profiler and accessibility overhead may affect absolute timings.

The renderer trace establishes blocking client computation during the observed stalls. It does not establish a streaming-specific regression, a memory leak, the full opening-delay breakdown, or the absence of additional transport, layout, compositor or system costs. Event rates, request/snapshot timings, the first conversation frame, a controlled comparison and a deterministic replay remain unmeasured.

Requested investigation and validation

Start at workEntryIndicatesToolFailureFromOutput and its timeline visibility caller. Record entry kind, relevant string lengths, invocation count, branch and duration without recording output text. Determine whether cost comes from repeated scans, a particular input or branch, or both.

In the inspected checkout, structured output scans are bounded at 32,768 characters, while the final detail/command scan is uncapped. This is a candidate to measure, not an identified cause. Optimize or cache the measured work while preserving failure-status correctness and invalidation when output changes. Add a regression test at the timeline-derivation boundary that exercises the observed pattern.

For the remaining opening delay, mark the selection click, route change, snapshot request/subscription, first response/event arrival, decoding, projection/timeline preparation and first visible conversation frame. Compare first entry with repeated switches into the same thread. Record the Mac server version and connection mode.

Validate on Windows desktop with a comparably large thread during both naturally quiet and active intervals. Exercise thread opening and upward scrolling. Measure selection-to-first-visible-conversation and scroll/input frame latency separately, and compare classification-related long tasks before and after the fix. Preserve failure statuses, scroll anchoring, restoration and update visibility. Check applicable shared web/mobile paths if the fix changes shared code.

@Sovex666

A second Windows data point for @Sovex666's breakdown, measured just now (29 Oct, ~13:30 UTC) on the machine behind this issue. It runs a fork of T3 Code (based on a recent nightly) with Codex and Claude threads active, and far fewer MCP servers configured than in Sovex666's setup.

Same metric as above (private memory, read via Win32_Process.PrivatePageCount, read-only):

Whole server process tree: 101 processes, 8.5 GiB private. The server process itself: 1.65 GiB. All Electron processes together: 1.0 GiB.
One shared Codex app-server (up for 4.8 h): 75 descendants, 4.8 GiB private. Of that, 25 python.exe processes hold 4.3 GiB.
Its direct children are five identical sets of MCP helpers, one set per Codex session: a Python MCP server (~800 MB private each), a run_mcp.py wrapper, a node and a node_repl helper. The five sets were started 5, 18, 60, 64 and 258 minutes earlier.

So the shape matches Sovex666's: the memory sits in per-session MCP helpers under the shared Codex app-server, not in the renderer. It scales with the number of sessions times the number of configured servers, and one heavy Python server is enough to make each extra session cost ~1 GB here.

What I could not establish: whether the 60-, 64- and 258-minute-old sets belong to sessions that were still in use. In that window the app had 3 Codex runs in progress and 43 that finished within the last hour, and I did not map helper sets to threads. So this does not show whether cleanup is late or the sets are being held as intended (#21905). A second app-server in the same tree, up for 88 h, had only 3 descendants and 60 MB, so idle app-servers do not accumulate helpers here.

Posted by Claude (Opus 5.5, Claude Code via T3 Code) on behalf of @AdEx-Partners-DE.

Navigation Menu

Repository navigation

T3 Code Nightly reaches approximately 11.5 GB while concurrent agent progress stalls #18485

What happened

Diagnosis

Steps to reproduce

Version

Environment

Evidence

Related issues

Fix applied or workaround

Filed by

Windows desktop: slow thread opening and scrolling/navigation stalls

Issues and where they appear

Environment

Reproduction scenario

Affected thread workload

Captured renderer path

Supporting observations and measurement limits

Requested investigation and validation

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

T3 Code Nightly reaches approximately 11.5 GB while concurrent agent progress stalls #18485

Description

What happened

Diagnosis

Steps to reproduce

Version

Environment

Evidence

Related issues

Fix applied or workaround

Filed by

Activity

juliusmarminge commented on Oct 12, 2026

Sovex666 commented on Oct 12, 2026

AdEx-Partners-DE commented on Oct 12, 2026

Sovex666 commented on Oct 21, 2026

Sovex666 commented on Oct 28, 2026

Windows desktop: slow thread opening and scrolling/navigation stalls

Issues and where they appear

Environment

Reproduction scenario

Affected thread workload

Captured renderer path

Supporting observations and measurement limits

Requested investigation and validation

AdEx-Partners-DE commented on Oct 29, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions