CLI: subagents memory leak

What version of Codex CLI is running?

0.130.0

What subscription do you have?

Pro Lite

Which model were you using?

No response

What platform is your computer?

Apple Silicon

What terminal emulator and version are you using (if applicable)?

No response

Codex doctor report

What issue are you seeing?

Spawning many subagents causes the native codex process memory and thread count to grow rapidly. The CLI can become temporarily unusable and may eventually exit/crash.

This appears to happen inside the native codex process, not in the Node wrapper or external child processes.

Observation:

codex RSS grew from ~23 MB to 315 MB+
vmmap physical footprint reached ~363.7 MB
MALLOC_SMALL resident grew from ~29.8 MB to ~229.0 MB
malloc allocation count grew from ~79k to ~691k
thread count grew from 22 to 68

When the session becomes large enough, RAM leak worsens indefinitely, eventually emitting "codex(...) MallocStackLogging: can't turn off malloc stack logging because it was not enabled." and make system unusable. (See #17139 )

What steps can reproduce the bug?

Spawn subagents in a large session

What is the expected behavior?

Subagent fanout should have bounded memory growth.

Completed subagents should not remain fully live in memory indefinitely. Their event queues, session loops, thread resources, and model/tool state should either be drained, cold-unloaded, or released after completion while preserving enough metadata/results for the parent thread.

The CLI should remain responsive and should not exit/crash.

Additional information

Cause 1: v1 child event queue may not be drained

codex::spawn creates a per-session unbounded event channel:

session/mod.rs:482, async_channel::unbounded()
session/mod.rs:747, next_event() consumes from that receiver.

When a child thread is spawned, the thread manager consumes the initial SessionConfigured event, but it does not drain later child events:
thread_manager.rs:1197
thread_manager.rs:1245

The completion watcher only subscribes to child status, waits for a terminal status, and injects a completion message into the parent. It does not consume the child event stream:
agent/control.rs:320
agent/control.rs:943
agent/control.rs:1012
There is also TODO about this:
agent/control.rs:308

Cause 2: completed child sessions are not released
The completion watcher notifies the parent, but the child session remains live. Actual release appears to happen through the explicit close path:
agent/control.rs:715, shutdown_live_agent
agent/control.rs:737, close_agent
agent/control.rs:79, spawn slot reservation
agent/control.rs:99, release path

This means terminal children can keep their session loop, event channel, metadata, and associated runtime allocations alive until the user or client explicitly closes them.

Subagents getting the whole session transcript

Potential duplicates detected. Please review them and close your issue if it is a duplicate.

Codex Desktop: shutdown/nonexistent subagents appear to count toward spawn_agent thread limit #23219

Powered by Codex Action

Confirming repeated native codex OOM kills on Linux with Codex CLI 0.144.1 and 0.144.3.

Retained kernel journals contain three events in which the kernel recorded constraint=CONSTRAINT_NONE, global_oom, and killed the native codex process with SIGKILL:

Confirmed event	CLI/session state	Cumulative Codex agent/session tree	Rollout set on disk	Native `codex` anonymous RSS
1	0.144.1	34 nodes	154.1 MiB	3.42 GiB
2	0.144.1	72 nodes	345.6 MiB	3.35 GiB
3	0.144.3 process after resuming root state created on 0.144.1	155 nodes	1,022.6 MiB	2.84 GiB

The final two events belong to the same root session across a restart/resume. The tree counts are cumulative root, child, and nested session nodes, not concurrent workers.

At the three events, the host had 11.6 GiB usable RAM, no swap, approximately 3.9–4.1 GiB of shared/tmpfs memory, and approximately 0.10–0.15 GiB free. The kernel OOM task tables showed codex as the largest individual anonymous-memory process. Its oom_score_adj was 0.

The shell printed only Killed codex. Codex logs ended approximately 15 seconds, 4 minutes 35 seconds, and 14 seconds before the corresponding kernel records. I found no Rust panic, backtrace, fatal Rust error, clean-shutdown marker, or Codex coredump for these events.

Environment: Linux x86_64, GNU Screen 5.0.1, multi-agent CLI workload.

Sanitized codex doctor --json summary:

{
  "schemaVersion": 1,
  "codexVersion": "0.144.3",
  "overallStatus": "ok",
  "statusCounts": {
    "ok": 18
  }
}

No raw doctor output, logs, rollout files, terminal snapshot, process listings, or log database are attached because they contain local paths, project content, and session metadata.

Another Linux data point, with kernel-side evidence for how bad this gets when MCP servers are configured.

Environment

codex-cli 0.144.3 (npm, x86_64 musl vendor build)
Linux 6.17, 128 GB RAM, 512 MB swap
3 MCP servers in config.toml: 2 stdio (one of them loads a local embedding model), 1 streamable-http

What happened

A research fan-out with 12 parallel codex exec subagents (read-only, web/GitHub research). Within ~10 minutes the box allocated ~30-40 GB and went into a global memory livelock. It was unreachable for 8 h 35 min, then recovered on its own.

This matches the mechanism described in #23749: each subagent receives the full parent transcript and starts its own complete MCP stack. With K subagents that is K × the entire stack, including K copies of an embedding-model-backed server.

Timeline (from sar, 10-minute sampling)

time	mem avail	page cache	load-1	blocked	majflt/s	pgpgin/s
12:00	45.3 GB	31.0 GB	0.94	0	0.00	0.7
12:10	39.8 GB	19.7 GB	1.16	0	0.23	1,684
12:20	2.06 GB	4.0 GB	11.6	64	213	59,797
12:40	1.94 GB	3.9 GB	674	87	4,044	1,196,746
21:10	29.3 GB	7.3 GB	0.67	0	24.7	11,144

sadc could not write a single sample between 12:40 and 21:00 despite a 10-minute timer.

Independent confirmation of the freeze duration

An unrelated container logged a line whose application-internal timestamp is 12:25:42, while the Docker-side write timestamp is 21:00:06. The process was blocked for 8.5 hours before it could flush one line to stdout.

Why the OOM killer took 8.5 hours

The kernel kept reclaiming page cache instead of killing anything (cache is always nominally reclaimable), and with only 512 MB of swap there was nowhere to put anonymous memory. Cumulative PSI full memory stall was ~8.95 h. It eventually fired:

oom-kill:constraint=CONSTRAINT_NONE,...,global_oom,task_memcg=/user.slice/...
Out of memory: Killed process <pid> (MainThread)

The moment the global OOM killer ran, the machine recovered. So this is not only "codex uses a lot of memory" - with no cgroup limit on the user session it degrades into a whole-host livelock rather than a failed codex run.

Peak of the cgroup the fan-out ran in: 64.5 GiB.

Workaround

Disabling all MCP servers for fan-out runs:

codex exec \
  -c mcp_servers.<name>.enabled=false \
  ... (one flag per server)

Research subagents do not need local MCP context anyway.

Note on a related footgun: -c mcp_servers="{}" looks like it should clear the table but is a no-op on 0.144.3 - codex mcp list -c mcp_servers="{}" still reports every server as enabled. Only the per-server enabled=false form works. A profile file layered via --profile also did not toggle them in my testing. Happy to open that separately if it is not already tracked.

Suggestion: sharing one MCP stack across subagents of the same run, or at minimum a documented per-run switch, would remove the K× multiplier that makes this dangerous on machines that otherwise have plenty of RAM.

tui: fix unbounded memory growth from background agent event buffers

tui: free event buffers for closed background agent threads

Hit this at 17.3 GB RSS after a 32 hour multi-agent CLI session. Root cause:

Every observed thread gets a lazily created ThreadEventChannel (tui/src/app/thread_routing.rs) that buffers notifications while the thread isn't the active view. Channels for closed background agents are never removed. The buffer is also bounded by entry count (THREAD_EVENT_CHANNEL_CAPACITY = 32768, raised from 1024 in #9957), not bytes, and buffered ItemCompleted notifications carry full payloads (aggregated_output is ~1MiB, MCP results can be bigger). So a finished subagent can leave gigabytes pinned until the session ends.

vmmap on the 17.3 GB process shows 12.6 GB of MALLOC_LARGE across 1115 allocations (~11 MB average), which lines up with retained buffered payloads. Probably also behind #23749, #24668 and #20740.

I've been running this fix locally: free the channel when a background thread emits ThreadClosed (switching to it later just backfills from server history), plus a smaller capacity. Patchy@97dccc8fa5 if useful.

This matches what I hit. In my run the retained memory was dominated by buffered ItemCompleted notifications from finished background codex exec subagents — the tool-call results in those payloads are the largest contributor, well past the ~1 MiB aggregated_output, so an entry-count cap rather than a byte cap let a handful of closed agents pin most of the resident growth.

Both parts of your fix look independently useful: freeing the channel on ThreadClosed stops retention from scaling with the number of finished subagents, and the 32768 → 4096 cap bounds the live-thread case where a single busy agent buffers large results.

Any chance you'd open it as a PR against openai/codex? It currently only lives in your fork, and #23749 / #24668 / #20740 look like the same root cause.

@etraut-openai

Thanks for confirming. I'd happily open a PR but contribution rules say PRs are invite only (docs/contributing.md), uninvited ones get closed. i see @etraut-openai triaged this issue before, so flagging the root cause analysis above. The change is contained to the TUI app layer (thread_routing.rs plus one constant). Also the const change to THREAD_EVENT_CHANNEL_CAPACITY is more of a arbitrary suggestion i'm just running with.

Hit the same issue with the ChatGPT/Codex desktop app.

Slim fork to the event-buffer leak fix plus local build infrastructure

Additional observation on macOS Apple Silicon with Codex CLI 0.162.1, after a long-running multiagent coding session. The user explicitly authorized this report.

The user observed a native Codex process consuming about 32 GB and restarted it. The old process/PID and heap were not captured, so the 32 GB figure is a user observation, not an independently profiled measurement. Restarting allowed work to resume. A deterministic minimal reproduction and the allocation source remain unconfirmed.

Measured context, using bounded read-only diagnostics after restart (2026-10-15 UTC):

macOS 15.7.7, build 24G720, Apple Silicon; installed CLI and shared app-server daemon are 0.162.1. The long-lived thread was originally created under 0.162.0 and uses paginated history.
The affected working conversation's on-disk rollout was 397,129,477 bytes (about 379 MiB), with 43,374 indexed history items and a latest item ordinal around 154,639. These are metadata counts; no transcript content is attached.
The shared thread-history database was 3,771,887,616 bytes; this is the total database across conversations, not the size or RAM usage of this one thread.
Apple's footprint utility, which accounts for compressed/swapped memory better than RSS alone, measured the newly started native CLI at 56 MB, peak 139 MB. The surviving shared daemon was 1,034 MB, peak 2,230 MB. Another older native client was 1,676 MB, peak 1,765 MB. These post-restart observations do not reproduce or explain the terminated process's 32 GB.
Recent bounded Codex log inspection did not reveal an explicit allocator/OOM failure. No heap dump, allocation trace, or memory-growth time series of the old process is available.

The workload involved multi-day coding, repeated tool calls and CI polling, and several subagents over the conversation's lifetime. Completed agents were still listed as completed. Their exact history allocation and retention during the incident was not measured. Most recent local activity was small JavaScript orchestration and metadata reads, with Rust compilation confined to hosted CI.

This appears relevant to this issue's symptom, but I am not claiming that its original 0.130.0 event-queue/lifecycle explanation is verified for this incident or for 0.162.1. The installed-version source has since been reorganized, including residency/runtime modules. Potential areas for instrumentation are child-session residency and queued event bytes after completion, retained TUI/history render data despite pagination, and tool/diff payload retention. These are hypotheses to test, not established causes.

No credentials, local user paths, raw conversation text, repository source, or raw diagnostic logs are included. We have retained the session rather than deleting history or changing global settings.

Public-source follow-up to the incident report above; this adds no local diagnostics or system measurements.

A potentially relevant upstream change is #51761, which reduces tool-catalog metadata copying in the code-mode runtime.

I checked the published release sources:

0.162.1 globals.rs still clones state.enabled_tools in both tool-installation paths (lines 64 and 81).
0.163.0 globals.rs removes those two clones. The PR also narrows retained callback state to tool names and kinds.
GitHub commit comparisons show that merge commit 3e544e843713d254dfc22ced37b902235a7573f8 is an ancestor of rust-v0.163.0, while rust-v0.162.1 diverges before it. Direct source inspection confirms the relevant difference rather than relying on ancestry alone.

This is an investigation lead, not a confirmed explanation for the reported memory growth. Without an affected-process heap profile or a controlled reproduction, I cannot attribute the incident to this copying behavior or claim the newer release resolves it.

Sanitized structural follow-up to the memory incident reported above. A read-only scan of the long-running multi-agent session produced these aggregate counts on 2026-10-16:

JSONL size at sampling: approximately 416 MiB; 171,697 records.
Largest individual record: 950,164 bytes (less than 1 MiB).
response_item: 69,496 records, approximately 208 MiB.
event_msg: 71,578 records, approximately 129 MiB.
compacted: 159 records, approximately 60 MiB.
token_usage_record: 17,126 records, approximately 15 MiB.
Subtype event_msg/item_completed: 48,598 records, approximately 111 MiB.
Subtype response_item/reasoning: 19,602 records, approximately 100 MiB.
Subtype response_item/custom_tool_call_output: 10,511 records, approximately 59 MiB.
Subtype response_item/agent_message: 12,965 records, approximately 18 MiB.

These counts may help select a long-history, repeated-compaction and multi-agent workload for retained-state profiling. They do not establish the original 32 GB process growth's root cause; no affected-process heap profile or controlled reproduction is available. The earlier public-source lead concerning tool-catalog metadata copying remains unconfirmed.

No conversation text, reasoning text, tool-output contents, paths, usernames, credentials, session identifiers, process identifiers, raw logs or databases are included in this report. Only the aggregate structural counts and byte totals above are shared.

tui: bound per-thread replay buffers so long multi-agent sessions sto…

tui: add bounded_thread_replay to stop long multi-agent sessions growing

More data, plus a root cause on current main (8c07694). I gathered it from outside the running processes: ps/top, the rollout files, and reading the source. Nothing was attached to the processes.

Workaround: quit the TUI and run codex resume <session-id>. The same session came back at 79 MB, down from 13 GB. That frees 15 GB of swap, and the session state stays intact.

Observations (codex-cli 0.162.1/0.163.0, macOS arm64, `multi_agent_version = v2`, `max_concurrent_threads_per_session = 12`)

TUI process	uptime	memory (`top` MEM)	resident (RSS)
A: root + 600 v2 subagents	3.4 days	13 GB	0.2 GB
A after quit + `codex resume`	minutes	79 MB
B	4.2 days	4.0 GB → 5.3 GB within 1.5 h	0.15 GB

Almost all of it is compressed or swapped out: memory that is retained but never touched.

Session A's rollouts:

1 root thread and 600 subagent threads (spawn_agent 43, followup_task 1767, send_message 2533, close_agent 0);
root rollout: 446 MB, ~163k lines, 105 compactions;
subagent rollouts: ~1.5 GB in total.

/export of this session froze the TUI for a long time and wrote a 121 MB markdown file. export_transcript runs inline on the UI task: it hydrates the complete history, clones every item, builds every history cell, and only then assembles one string.

Root cause (TUI side)

The in-process app server attaches the TUI connection to every created thread (app-server/src/in_process.rs, thread_created_rx → try_attach_thread_listener). That includes every v2 subagent. The TUI never calls thread/unsubscribe for subagents during a normal session.
For each thread the TUI keeps a ThreadEventStore replay buffer (tui/src/app/thread_routing.rs ensure_thread_channel). The buffer is capped only at THREAD_EVENT_CHANNEL_CAPACITY = 32768 events. The only byte cap covers agent-message deltas (256 KiB, thread_event_buffer.rs). Completed items with full command output and reasoning, and every TurnDiffUpdated (the whole aggregated turn diff, re-sent after each edit), are kept in full. Replay ignores TurnDiffUpdated anyway: on_turn_diff only logs it.
Channels are removed only on reattach, prompt revert and side-thread discard. ThreadClosed only marks the picker entry closed.
Core memory stays bounded. Idle v2 children are evicted when residency capacity is reached (core/src/agent/control/residency.rs). That eviction does not emit thread/closed: the listener just ends. The idle-unload path that does emit it (app-server/.../thread_lifecycle.rs) requires no subscribers, and the TUI is always subscribed (point 1). So the TUI keeps up to 32768 events for each of 600+ threads for the whole session.

Freeing buffers on ThreadClosed (proposed earlier in this issue) therefore doesn't help v2 sessions: in this flow ThreadClosed never arrives.

Proposed fix, behind an off-by-default feature flag

Commit 45b6d51 on https://github.com/konard/codex/tree/tui-bound-thread-replay-memory. It is TUI-only and adds [features] bounded_thread_replay = true (Stage::UnderDevelopment). With the flag off, nothing changes. With it on:

each per-thread replay buffer is also bounded by estimated serialized bytes (16 MiB);
turn/diffUpdated is no longer buffered;
when a persisted background subagent finishes a turn, or any persisted thread completes a context compaction, its buffer is compacted to what a history refresh can't rebuild (pending requests, hook and MCP status), and the thread is marked truncated. RAM then holds roughly "since the last compaction or idle point", not the whole session;
a truncated thread is reloaded from server history (thread/resume) when selected. Inferred sessions already use this refresh_snapshot_session_if_needed path.

This does the same thing as codex resume, but per thread and continuously.

Possible follow-ups:

App::transcript_cells for the displayed thread is also unbounded, and could be trimmed at compactions the same way;
/export could stream from paginated history off the UI task.

I know external PRs aren't accepted. The patch is there if it's useful.

Additional privacy-filtered observation on Apple Silicon, measured 2026-10-19 UTC:

OS: macOS 15.7.7, arm64.
The installed binary at the observed process executable path reports codex-cli 0.163.0. I have not independently established whether the long-running process is the same build as that binary.
A native codex process approximately 83 hours old had 18.8 GiB physical footprint (18.8 GiB peak) according to vmmap -summary.
Its writable mappings totaled 20.2 GiB, including 18.6 GiB swapped out; total mapped-region swap was 18.7 GiB.
The resident-memory process-list value was only about 127-153 MiB, so RSS alone substantially understated this process's memory pressure.
The machine had approximately 29.4 GiB swap in use and 5.5 GiB free disk space.
This occurred during sustained coding work with completed and active subagents across long-lived sessions. The operator had also reported an earlier Codex process around 32 GB and restarted it. That earlier observation is not independently measured here.

These are point-in-time observations, not proof of monotonic growth, an allocation root cause, or that subagents caused the retained memory. This is memory attributed to the native Codex process itself; no summed child-process memory or virtual address-space reservation is being presented as physical RAM.

No heap dump, conversation, prompt, repository content, credentials, username, filesystem path, raw process arguments, or session logs are included. Only version/OS and aggregate process-memory observations are shared.

Expected behavior: long-lived sessions and completed subagents should have bounded native-process retention; diagnostics should make swapped/compressed footprint visible when RSS is misleading. A targeted, content-free allocation-category diagnostic would help distinguish retained session/event queues from unrelated runtime allocations.

tui: add bounded_thread_replay to stop long multi-agent sessions growing

Navigation Menu

Repository navigation

CLI: subagents memory leak #23260

What version of Codex CLI is running?

What subscription do you have?

Which model were you using?

What platform is your computer?

What terminal emulator and version are you using (if applicable)?

Codex doctor report

What issue are you seeing?

What steps can reproduce the bug?

What is the expected behavior?

Additional information

Observations (codex-cli 0.162.1/0.163.0, macOS arm64, `multi_agent_version = v2`, `max_concurrent_threads_per_session = 12`)

Root cause (TUI side)

Proposed fix, behind an off-by-default feature flag

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

CLI: subagents memory leak #23260

Description

What version of Codex CLI is running?

What subscription do you have?

Which model were you using?

What platform is your computer?

What terminal emulator and version are you using (if applicable)?

Codex doctor report

What issue are you seeing?

What steps can reproduce the bug?

What is the expected behavior?

Additional information

Activity

github-actions commented on May 18, 2026

emilfihlman commented on Jul 14, 2026

fa1k3 commented on Jul 27, 2026

Patchy commented on Aug 11, 2026

fa1k3 commented on Aug 11, 2026

Patchy commented on Aug 11, 2026

dkorunda-tp commented on Sep 18, 2026

konard commented on Oct 15, 2026

konard commented on Oct 16, 2026

konard commented on Oct 16, 2026

konard commented on Oct 18, 2026

Observations (codex-cli 0.162.1/0.163.0, macOS arm64, multi_agent_version = v2, max_concurrent_threads_per_session = 12)

Root cause (TUI side)

Proposed fix, behind an off-by-default feature flag

konard commented on Oct 19, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Observations (codex-cli 0.162.1/0.163.0, macOS arm64, `multi_agent_version = v2`, `max_concurrent_threads_per_session = 12`)