Skip to content

Background tasks have no turn-timeout — a stalled first model request hangs the turn forever at task_started #489

Closed as duplicate

Description

@Markiss-88

Summary

A --background Codex task can hang indefinitely when the turn's first model request stalls (e.g. a transient token-refresh / network stall in the detached, headless context). There is no turn-timeout or stall-heartbeat, so the turn stays parked at task_started forever, then the detached process eventually dies silently — producing no output and no error to the caller.

What we observed

A background delegation stalled and never recovered. The evidence:

  • Job log stops at the turn boundary and never advances:

    Starting Codex Task.
    Queued for background execution.
    Starting Codex task thread.
    Thread ready (<thread-id>).
    Turn started (<turn-id>).
    

    …and nothing after that.

  • Rollout JSONL for the thread contains only the opening events and then stops:

    session_meta
    event_msg: thread_settings_applied  (model set, provider openai)
    event_msg: task_started
    

    No model response, no tool calls, no completion, no error event — the file simply stops growing.

  • The detached process is gone; the caller gets no result and no failure signal (it just waits).

It is not the model / auth / prompt

The same model + same prompt via foreground codex exec completes normally in ~40s. So the model, the ChatGPT-plan auth (token refreshes fine on-demand), and the prompt are all fine. The failure is specific to the background path lacking a timeout/recovery around the first model request.

Impact

Anyone delegating real work via --background can silently lose an entire task to a single transient stall, with no error to react to. A caller that just polls for completion waits forever.

Suggested fix

Add a turn-timeout or stall-heartbeat on background turns: if the rollout hasn't advanced past task_started (no new events) within N seconds, abort the turn with a visible error the caller can catch and retry — fail loudly and recoverably instead of hanging forever. A configurable timeout (e.g. background_turn_timeout_sec) would let callers tune it.

As a stopgap on the caller side, a stall can be detected externally by watching the rollout JSONL for growth: if the last event is still task_started / thread_settings_applied and the file hasn't grown for a few minutes, treat the task as dead and retry foreground. But a built-in timeout would be the right fix.

Environment

  • codex-cli 0.144.1
  • Claude Code Codex plugin, background (detached) task mode
  • macOS

Activity

  1. axisrow commented on Jul 14, 2026

    @axisrow

    +1, and the fix shape already exists two files over. The broker layer races async work against a bounded timer with cleanup on both paths (sendBrokerShutdown / waitForBrokerEndpoint in scripts/lib/broker-lifecycle.mjs — bounded timeout, unref/clear on the success path, failure-cleanup on breach). The same pattern applies directly to runTrackedJob: race runner() against an env-overridable deadline, and on breach write status: "failed" with an explicit timeout reason before process-exit handling — exactly what the issue specifies. The task_started-then-silence signature you describe is the cleanest detection signal: a stall-heartbeat keyed on rollout-JSONL growth would catch the transient first-model-request stall without a hard wall-clock cutoff. Either way, the current "hang forever, then die silently" is the worst failure mode for a detached worker.

  2. axisrow commented on Jul 20, 2026

    @axisrow

    This looks like a duplicate of #49 — same root cause (no turn-level timeout in captureTurn), same symptom (stalled model request hangs forever). #49 has a ready PR (#376 by @russjhammond) that wires the dead-code rejectCompletion to client.exitPromise and makes the turn budget configurable. Worth linking these so the fix gets visibility.

  3. dkundel-openai commented on Aug 11, 2026

    @dkundel-openai
    Collaborator

    Duplicate of #520.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    duplicateThis issue or pull request already exists

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions