Skip to content

Codex app-server repeatedly killed by SIGKILL after memory grows to ~27GB when handling large tool/log output #24048

Open

Description

@Yanhanghanghang

What version of the Codex App are you using (From “About Codex” dialog)?

Codex Desktop: please fill from "About Codex" VS Code Codex extension also installed: openai.chatgpt, version 26.519.32039 shown in VS Code Marketplace

What subscription do you have?

Please fill in your plan, e.g. ChatGPT Pro

What platform is your computer?

Darwin 24.5.0 arm64 arm macOS

What issue are you seeing?

Codex Desktop has repeatedly crashed today with:

(code=null, signal=SIGKILL)

The UI asks me to reload Codex. This happened multiple times on 2026-05-22, and once escalated into the whole computer becoming unstable/rebooting.

Observed error messages in the crash screen included:

(code=null, signal=SIGKILL).

Most recent error:
{"timestamp":"2026-05-22T07:26:01.761510Z","level":"WARN","fields":{"message":"could not find callback for Integer(1)"},"target":"codex_app_server::outgoing_message"}
{"timestamp":"2026-05-22T07:26:01.761563Z","level":"WARN","fields":{"message":"could not find callback for Integer(0)"},"target":"codex_app_server::outgoing_message"}

### What steps can reproduce the bug?

I do not have a minimal deterministic repro yet, but the bug happened repeatedly during a Codex crash investigation on 2026-05-22.

Observed pattern:

Run Codex Desktop on macOS.
Have the VS Code Codex extension installed/enabled at the same time, so both Codex Desktop and the VS Code extension may run app-server processes.
In a Codex session, investigate previous crashes or logs using commands that can produce large output, for example:
rg ~/.codex/sessions
/usr/bin/log show
ps -axo ...
Codex app-server memory grows very large.
Eventually the Codex UI shows: (code=null, signal=SIGKILL).
The UI asks to reload Codex. This happened multiple times in one day. In one case the whole computer became unstable and rebooted.

### What is the expected behavior?

Codex app-server should not be able to grow to tens of GB of memory due to tool/log output. Large command output should be capped, streamed, discarded, or backpressured before it is accumulated in app-server memory.

Suggested product-side fixes:

Add a hard output cap before app-server stores or serializes tool output.
Apply streaming discard/backpressure for terminal/log output.
Do not rely only on UI truncation; truncation should happen before backend memory accumulation.
Add memory budget isolation for tool output so one large command cannot kill the app-server.
Consider detecting high-risk commands that may emit unbounded output and require explicit confirmation or automatic output limiting.
If VS Code extension and Codex Desktop both run app-server processes, make this safer or clearer to users.

### Additional information

_No response_

Activity

  1. added
    bugSomething isn't working
    app-serverIssues involving app server protocol or interfaces
    tool-callsIssues related to tool calling
    on May 22, 2026
  2. jshaofa-ui commented on May 22, 2026

    @jshaofa-ui

    Proposed Solution: App-Server SIGKILL at ~27GB Memory

    Root Cause

    The app-server accumulates tool call results and log output in memory without:

    1. Streaming/truncation of large outputs
    2. Memory limits per session or per tool result
    3. Backpressure mechanisms when memory approaches limits
    4. Proactive memory monitoring

    Proposed Fix

    File: src/app-server/tool-executor.ts — Enforce tool result size limits:

    const TOOL_RESULT_MAX_SIZE = 1024 * 1024; // 1MB
    if (result.output.length > TOOL_RESULT_MAX_SIZE) {
      result.output = result.output.slice(0, TOOL_RESULT_MAX_SIZE) +
        `[Output truncated: ${result.output.length} bytes total]`;
      result.truncated = true;
    }

    File: src/app-server/session-manager.ts — Memory monitoring:

    async checkMemory(sessionId) {
      if (usage > this.memoryLimit * 0.9) await this.evictOldResults(sessionId);
      if (usage > this.memoryLimit * 0.95) await this.flushLogsToDisk(sessionId);
      if (usage > this.memoryLimit) throw new MemoryLimitError(...);
    }

    Key Changes

    • src/app-server/tool-executor.ts: Add result size limits (1MB default)
    • src/app-server/session-manager.ts: Memory monitoring + eviction
    • src/app-server/tools/read-file.ts: Stream large file reads
    • src/app-server/health.ts: Periodic memory health checks

    Full solution: solutions/codex-24048-app-server-sigkill-memory-fix.md

  3. jshaofa-ui commented on May 22, 2026

    @jshaofa-ui

    Proposed Solution: App-Server SIGKILL at ~27GB Memory

    Root Cause

    Unbounded memory accumulation across 5 paths:

    1. ThreadHistoryBuilder accumulates full turn history in memory
    2. Callback HashMap grows without bounds for pending callbacks
    3. WebSocket outbound channel buffers messages without backpressure
    4. PendingCallbackEntry holds references to large tool outputs
    5. ThreadState.current_turn_history retains complete history

    Fix

    1. Hard Output Cap: 256KB cap per tool call output
    2. Memory Budget Isolation: 4MB per-turn budget with eviction
    3. Callback HashMap TTL: 1024 entry limit + 5min TTL
    4. WebSocket Message Size Limit: 1MB per message
    5. High-Risk Command Detection: 64KB cap for cat, find, grep, etc.

    Key Code Change

    // codex-rs/app-server/src/tool_execution.rs
    fn cap_tool_output(output: String, max_bytes: usize) -> String {
        if output.len() > max_bytes {
            let truncated = &output[..max_bytes];
            format!("{}... [output truncated: {} bytes omitted]", truncated, output.len() - max_bytes)
        } else {
            output
        }
    }

    Full solution: solutions/codex-24048-app-server-27gb-sigkill-fix.md

  4. w2jmoe commented on May 27, 2026

    @w2jmoe

    This is a pretty important report ⚠️

    The key point here probably isn't just “large output”, but the absence of a hard execution/output boundary inside the app-server pipeline.

    Once tool output becomes effectively unbounded, the system stops behaving like a controlled runtime and starts behaving like a memory sink.

    Especially dangerous when:

    • multiple clients/extensions coexist
    • logs/grep outputs are streamed
    • backend buffering happens before truncation

    Really agree with this part:

    truncation should happen before backend memory accumulation

    UI-side truncation alone usually comes too late.

    The “tool output budget” idea also feels very important for long-running agent environments. Otherwise a single pathological command can destabilize the whole runtime.

    Good catch 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    appIssues related to the Codex desktop appapp-serverIssues involving app server protocol or interfacesbugSomething isn't workingperformancetool-callsIssues related to tool calling

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions