Skip to content

CLI/TUI session can restart during long tool-heavy turns after repeated 'Function call output is missing for call id' #14824

Open
Open
CLI/TUI session can restart during long tool-heavy turns after repeated 'Function call output is missing for call id'#14824
Labels
bugSomething isn't workingtool-callsIssues related to tool calling

Description

@drindt
drindt
opened on Mar 16, 2026

Summary

Codex repeatedly restarts or loses turn continuity during long-running sessions with many tool calls,
especially while polling active exec_command sessions via repeated write_stdin calls.

The visible user symptom is that the agent "crashes after a while", the conversation resumes with
lost short-term continuity, and in some cases running command output retrieval gets aborted mid-flow.

Environment

  • Codex client version: 0.114.0
  • Observed on: 2026-03-16
  • Host path with logs: /home/drindt/.codex
  • Active thread during the most recent reproductions:
    019cf764-6fc4-7cd3-81d3-3872977ee047

User-visible behavior

  • Long-running task proceeds normally for several minutes.
  • Agent polls running commands repeatedly with write_stdin.
  • After a while the session appears to restart or lose continuity.
  • The user sees repeated interruptions and has to say Fortsetzen. multiple times.

Reproduction pattern

  1. Start a long-running terminal task with exec_command, for example:
    make gcp-development-vm-tunnel-up
  2. Poll the running session repeatedly with write_stdin.
  3. Interleave further tool calls and file edits during the same long turn.
  4. After enough iterations, Codex loses continuity and the session is effectively restarted.

Strong indicators from local logs

Repeated internal normalization error

/home/drindt/.codex/log/codex-tui.log shows repeated entries like:

  • 2026-03-16T17:45:09Z ... Function call output is missing for call id: call_8RF9wsbJahTQ8eOYUnuYLKYa
  • The same message repeats many times for the same thread.
  • A second call id also appears later:
    call_G7bB0oUZRjUNo8KOmxuv8oAU

This is visible both in codex-tui.log and in logs_1.sqlite.

Session restart / shutdown around the same time

The same log file shows session shutdown/re-init around the affected period:

  • 2026-03-16T17:52:41Z ... codex_core::codex::handlers: Shutting down Codex instance
  • Shortly after:
    2026-03-16T17:52:54Z ... Resumed rollout successfully from "/home/drindt/.codex/sessions/2026/03/16/rollout-2026-03-16T17-04-40-019cf764-6fc4-7cd3-81d3-3872977ee047.jsonl"

Related warnings seen in the same timeframe

  • Shell snapshot deletion warnings:
    Failed to delete shell snapshot ... No such file or directory
  • File watcher warning:
    failed to unwatch /home/drindt/.codex/skills/.system: No watch was found
  • MCP process group cleanup warnings in earlier restarts:
    Failed to kill MCP process group ... No such process

These warnings may be secondary, but they cluster around the restart events.

Concrete evidence query

The following query returns the relevant thread-local events:

select datetime(ts,'unixepoch','localtime'), level, target, substr(message,1,220)
from logs
where thread_id='019cf764-6fc4-7cd3-81d3-3872977ee047'
  and ts >= strftime('%s','2026-03-16 17:40:00')
order by ts desc, ts_nanos desc
limit 120;

Likely failure mode

Codex appears to enter an inconsistent internal state where tool-call bookkeeping loses the output for
one or more call ids (Function call output is missing for call id ...). After that, the session is
eventually shut down and resumed, which looks like a crash from the user perspective.

This does not currently look like a user-shell process crash in the target repo. It looks more
like an internal Codex session/state management bug during long tool-heavy turns.

Impact

  • Long debugging sessions become unreliable.
  • Operator trust drops because the agent appears to "randomly crash".
  • The user must manually continue the session multiple times.
  • Mid-flight reasoning context is partially lost even though rollout resume exists.

Requested investigation

  1. Investigate why context_manager::normalize repeatedly logs
    Function call output is missing for call id ....
  2. Check whether repeated write_stdin polling of long-running exec_command sessions can orphan or
    desynchronize tool-call bookkeeping.
  3. Check whether session shutdown/resume is being triggered as a recovery path for this state.
  4. Review the surrounding shell snapshot and file watcher warnings for causal relevance.

Relevant local files

  • /home/drindt/.codex/log/codex-tui.log
  • /home/drindt/.codex/logs_1.sqlite
  • /home/drindt/.codex/history.jsonl

Activity

  1. github-actions commented on Mar 16, 2026

    @github-actions
    github-actionsbot
    on Mar 16, 2026 – with GitHub Actions
    Contributor

    Potential duplicates detected. Please review them and close your issue if it is a duplicate.

    Powered by Codex Action

  2. etraut-openai
    added
    bugSomething isn't working
    tool-callsIssues related to tool calling
    on Mar 16, 2026
  3. kram3ko
    added a commit that references this issue on May 9, 2026

    feat(tg): orphan-tool-call auto-recovery + SIGTERM-safe shutdown

    efba5f8

  4. dogancanbaris commented on Jul 10, 2026

    @dogancanbaris
    dogancanbaris
    on Jul 10, 2026 – with ChatGPT Codex Connector

    I can provide a closely matching Windows Codex Desktop reproduction with local SQLite and rollout evidence. This also appears related to #26306 (dramatic quota/context regression) and #19571 (Pro 5-hour depletion).

    Environment

    • Product: Codex Desktop on Windows x64
    • App package/version: 26.707.3748.0
    • App-server/session runtime: 0.144.0-alpha.4
    • Standalone CLI installed: codex-cli 0.144.0
    • Subscription: ChatGPT Pro
    • Model on the primary affected thread: gpt-5.6-sol, reasoning high

    Exact affected identifiers

    Primary affected thread:

    • Thread ID: 019f4865-f699-7152-af0b-6c3ebc3b5923
    • Spawned child ID: 019f48b3-cc9f-7260-b045-28b1d0a3da2c
    • Missing call ID repeatedly reported: call_NNIPL5C4X9xpf17lvuWcjO0o

    Other independently active root threads showing the same context-replay amplification:

    • 019f48ac-e788-7ff2-bbb1-692f809b9b70
    • 019f4885-b204-7de1-bcc2-3165e6d1ca1d
    • Earlier archived thread: 019f4888-e012-7e12-a091-aa5fc3fb4fc7

    OpenAI should be able to correlate these IDs with server-side request and quota telemetry.

    Exact failure

    From logs_2.sqlite, the primary thread logged this normalization/state error 198 times between approximately 2026-07-09 22:43:50 and 23:25:10 America/Toronto:

    Custom tool call output is missing for call id: call_NNIPL5C4X9xpf17lvuWcjO0o

    The error target was codex_core::util during run_turn.prepare_sampling_request_input. The same call ID repeated while the thread's token counter continued advancing.

    Usage/context evidence

    Primary thread 019f4865... at the diagnostic snapshot:

    • More than 243.7M cumulative local tokens
    • 1,234 token-count/model-continuation events
    • Approximately 241.9M input tokens
    • 98.5% of input reported as cached
    • Approximately 3.5M uncached input
    • Average request input approximately 196k tokens
    • Recent request inputs approximately 303k tokens
    • Model context window reported as 353,400
    • 6 compactions
    • 895 exec continuations
    • 318 wait continuations

    The 318 wait continuations alone accounted for approximately 65.6M cumulative input tokens because each continuation carried/replayed a context averaging roughly 200k tokens.

    Corroborating roots:

    • 019f48ac...: approximately 124.9M local tokens, 742 continuations, 96.4% cached, 5 compactions, 607 exec + 100 wait. This rollout also retained about 88 MB of base64 image tool outputs and spawned five workers from a large parent context.
    • 019f4885...: approximately 76.7M local tokens, 421 continuations, 97.4% cached, 3 compactions, 363 exec + 43 wait.
    • Archived 019f4888...: approximately 170.5M local tokens, 909 continuations, 98.1% cached, including 305 wait continuations accounting for about 58.1M input tokens.

    The cache itself appears to be working (96-98.5% hits). The pathological behavior is full-context replay across hundreds of small execution/polling continuations. The independent root counters total well over 600M cumulative input tokens, while uncached input across the four was only about 13M. Spawned-thread displayed totals were not naively added because they inherit parent history.

    During one roughly four-minute observation window, the local rate_limits telemetry moved from about 20% to 23% used for the five-hour bucket and 22% to 23% used for the weekly bucket while the affected roots continued running.

    No active agent_jobs row or separate hidden Codex CLI worker explained the drain. No active background memory-generation sampling job was observed during this diagnostic. The dominant live sampling was attached to these long tool-heavy threads.

    Suspected combined failure mode

    1. Long tool-heavy turns generate hundreds of model-backed polling/execution continuations.
    2. Each continuation replays a 150k-300k context even when almost entirely cached.
    3. Repeated polling eventually desynchronizes tool-call bookkeeping.
    4. Custom tool call output is missing... repeats for the same call ID.
    5. The affected thread remains live and continues advancing token/quota counters instead of failing closed or recovering once.

    Could maintainers investigate:

    • whether repeated wait/poll continuations can orphan custom-tool outputs in code-mode/app-server;
    • whether this normalization error causes retry/replay amplification;
    • why a stuck turn continues to advance usage rather than terminating;
    • whether cached full-context continuations are being weighted as intended against Pro limits;
    • and whether polling can be handled without a fresh full-context sampling request each time?

    I can provide further sanitized SQL queries or event timestamps if useful, but have intentionally omitted repository contents, user prompts, local paths, and credentials.

  5. ELUNIA-Aeon commented on Jul 29, 2026

    @ELUNIA-Aeon
    ELUNIA-Aeon
    on Jul 29, 2026 – with ChatGPT Codex Connector

    I observed a related persistence problem in Codex Desktop on Windows with bundled runtime 0.146.0-alpha.3.1.

    One quarantined thread contains 21 stable unmatched calls: 3 function_call and 18 tool_search_call records without corresponding outputs. The gaps remained present across read-only inspections. The thread could complete later turns while already containing those gaps, but after a later reload/resume, new message submission failed before a new user message or turn/started record was persisted.

    This may indicate that context reconstruction is where older incomplete call/output state becomes fatal, rather than necessarily at the moment the gap is first written. I did not modify the rollout or synthesize missing outputs.

    Separately, I also observed the same agent loop died unexpectedly UI symptom in structurally complete threads, so incomplete call/output history appears to be one cause, not the only cause. Related desktop/subagent report: #23971.

    Requested regression coverage:

    • persist every supported call type and its output atomically or recoverably;
    • reconstruct a thread containing an interrupted call without permanently killing later turn submission;
    • distinguish structural-history failure from process-local lifecycle failure;
    • report whether a failed submission was delivered, not delivered, or is uncertain.

  6. nemoriko commented on Aug 12, 2026

    @nemoriko
    nemoriko
    on Aug 12, 2026

    Another reproduction on Windows with CLI 0.147.0 — three silent session deaths in one day, all matching the "function call output is missing" fingerprint.

    Environment: codex-cli 0.147.0 (TUI), Windows 11 Pro (build 26200), PowerShell 7. Thread 019ff49e-b0b4-7830-8690-18cedaf5cecd, rollout rollout-2026-08-12T15-17-48-*.jsonl.

    Pattern (all times UTC, 2026-08-12)

    1. 11:07:16 — code-mode tools.shell_command starts an 11-file pytest run → "Script running with cell ID 308" after ~11s. 11:07:30 — model calls wait(cell_id=308, yield_time_ms=60000). The rollout contains the wait function_call record but no function_call_output ever follows; nothing further is written. The 60s yield deadline passed silently — the session was dead within 60s of the wait.
    2. After relaunch + resume: same pytest at 11:09:54 (cell 1), same wait at 11:10:08, same silent death within ~50s.
    3. Third death without any wait: 11:37:31, a short single-file pytest exec (timeout_ms=10000) — the rollout records the tool call, then nothing. Long-running wait is therefore not a necessary condition.

    Counter-example: an identical wait at 11:05:19 returned normally in 27.6s (the script had exited early) and the session continued.

    Local evidence: the internal log DB (logs_2.sqlite) shows three different process UUIDs across the window; the dying processes leave no WARN/ERROR at the moment of death. No Windows Event Log (WER) entries and no crash dumps — silent at the OS level too. Separately, the post-resume process logged a ~941-message burst of app-server event consumer lagged / consumer queue is full; dropping in-process app-server event warnings within one second while handling a ~40KB truncated tool output (that process survived).

    The pytest suite in question is fully mocked unit tests (no real process spawning or killing). Happy to provide sanitized rollout excerpts or run /feedback from a repro session if useful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingtool-callsIssues related to tool calling

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions