Skip to content

[Bug]: One unresponsive provider holds all four effect workers and stalls every thread #16636

Open

Description

@3mdistal

Before submitting

  • I searched existing issues and did not find a duplicate.
  • I included enough detail to reproduce or investigate the problem.

Area

apps/server

Steps to reproduce

  1. Make one provider stop answering control requests without exiting. Ours was the shared Codex app-server on Windows after it ran out of blocking threads ([Bug]: Codex app-server keeps per-thread MCP servers running after threads settle #15979, [Windows] Idle local STDIO MCP servers can exhaust Tokio blocking pool and stall the app-server openai/codex#42740); a stub provider whose startTurn never resolves should do the same.
  2. Send messages to four or more threads on that provider.
  3. Send a message to a thread on a different provider, such as Claude.

Expected behavior

The Claude thread starts. A stalled provider fails or times out its own effects without holding up the others.

Actual behavior

Every thread stalls, on every provider. Each unanswered provider-turn.start (and later provider-turn.interrupt) holds one of the orchestration effect workers, and there are four (DEFAULT_EFFECT_WORKER_CONCURRENCY in orchestration-v2/EffectWorker.ts). Each claim has a 30-second lease, but an expired lease isn't a deadline: the effect keeps running, no worker frees up, and Claude launches, checkpoints, and deliveries wait in orchestration_v2_effect_outbox behind it. At the first lockup, four effects (three provider-turn.start, one provider-turn.interrupt) had been running for 20 to 40 minutes, long past their leases, with eleven pending behind them: five starts, four checkpoints, a steer, and a title generation. Recovery needed restarting the Codex app-server once and the T3 backend once, which interrupted every in-flight run.

Impact

Blocks work completely

Version or commit

0.0.46-nightly.20261005.2702 (first lockup) and 0.0.46-nightly.20261006.2735 (second)

Environment

Windows 11, T3 Code desktop, codex-cli 0.160.1 as the shared Codex app-server, Claude as a second provider

Logs or stack traces

orchestration_v2_effect_outbox at 12:40 PM, first lockup:
  running  provider-turn.start      claimed 12:00:19  attempt_count 1  lease long expired
  running  provider-turn.start      claimed 12:01:04  attempt_count 1  lease long expired
  running  provider-turn.interrupt  claimed 12:19:26  attempt_count 1  lease long expired
  running  provider-turn.start      claimed 12:19:40  attempt_count 1  lease long expired
  pending  11 more: 5 starts, 4 checkpoints, 1 steer, 1 title generation
Codex provider log: turn/start, thread/start, turn/interrupt, thread/resume sent, never answered;
the app-server stayed alive and kept sending notifications.
For comparison, provider-turn.start normally finishes in p50 6.4 s, p95 26 s, max 33 s.

Suggested fix: give provider control effects an execution deadline that fails the run, and run each provider's effects on its own workers, so one wedged provider can't take every worker.

Activity

  1. juliusmarminge commented on Oct 7, 2026

    @juliusmarminge
    Member

    Note

    Grok responding on behalf of Julius.

    Confirmed on current main (517188b). One provider that stops answering can occupy every orchestration effect worker.

    Root cause.

    • Fixed pool of four workers.

      • apps/server/src/orchestration-v2/EffectWorker.ts:781 sets DEFAULT_EFFECT_WORKER_CONCURRENCY = 4.
      • runDaemonWithOptions (:784-853) starts that many identical runWorker loops, and layerDaemon (:857-858) uses the default.
      • The loops are not partitioned by provider. Per-thread ordering in EffectOutbox.ts:314-353 (claimableCandidatePredicate) only stops two effects on the same thread from running together, so the four slots are shared by every thread on every provider.
    • No execution deadline. runOnce (EffectWorker.ts:681-686) awaits Effect.raceFirst(executor.execute(...), cancellation). The only other branch is an explicit cancellation, with no timeout.

      • provider-turn.start → ProviderTurnStartService.start and provider-turn.interrupt → ProviderTurnControlService.interrupt (dispatch at EffectWorker.ts:152-204) add no timeout either. ProviderTurnControlService.ts:194-207 has a 2s deadline, but it only covers the poll after interruptTurn returns.
      • In the Codex adapter, client.request("turn/start", ...) (CodexAdapterV2.ts:5890) and the thread/start / thread/resume requests (~:6079-6110) are unbounded.
      • packages/effect-codex-app-server/src/client.ts has no default request timeout.
      • Only some interrupt sub-steps are bounded: timeoutOption("10 seconds") around waiting for completion (:6550, :6624). The bare turn/interrupt request at :6561 is not.
    • The lease is never enforced while the process is alive.

      • claimNext (EffectOutbox.ts:518-552) writes lease_expires_at but only claims status = 'pending' rows (:324).
      • Nothing compares lease_expires_at with the current time, and there is no renewal or reclaim of expired leases.
      • Expired running rows are only handled by reconcileAfterProcessLoss (:483-512) on the next server start.
      • workerId is per process (EffectWorker.ts:538), so the ownership checks in succeed/fail/retry still pass long after the lease has expired.

      In practice the 30s lease is metadata only. A fiber blocked on a silent provider holds its slot until that provider answers or the server restarts, which matches the outbox snapshot in the report.

    Suggested fix direction. Both parts from the report seem right, and they work independently:

    1. Give provider-control effects an execution deadline that fails the effect on timeout. Either raise it above the observed p95/max for provider-turn.start (~33s) or derive it from the lease and renew the lease while the call is healthy. On timeout, interrupt the fiber so its slot frees up, and settle the run through the existing retry/fail path.
    2. Partition worker capacity, either with per-provider (or per-provider-instance) lanes or a per-provider cap on running effects inside the claim query. That way one wedged provider can't take every slot.

    A unit test with a stub executor whose provider-turn.start never resolves, plus four or more threads, would reproduce this deterministically.

    Related.

    I didn't find an open PR that adds a deadline or per-provider lanes.

  2. added
    bugSomething is broken or behaving incorrectly.
    via-triageFiled through npx t3 triage
    on Oct 7, 2026
  3. krivenkop commented on Oct 18, 2026

    @krivenkop

    Note

    Claude Opus 5.5, an AI agent in Claude Code, writing on behalf of @krivenkop. Diagnosed from statev2.sqlite and the server log on their machine.

    Another occurrence on 0.1.0 (Linux, t3 serve), and in our case no Codex involved: every stuck effect was on a Claude Agent provider instance (through a local Anthropic-compatible proxy that kept answering normally; it served only 46 short requests during the stall, all 200 under 10 s).

    Timeline (UTC, 2026-10-18), from orchestration_v2_effect_outbox:

    created completed min effect status
    00:32:44 01:41:03 68 provider-turn.start (thread A) cancelled
    00:41:29 03:11:41 150 provider-turn.start (thread B) cancelled
    00:42:10 03:37:32 175 checkpoint.capture (thread C) succeeded
    00:47:12 03:17:53 150 provider-turn.start (thread D) cancelled
    01:41:06 02:13:15 32 provider-turn.start (thread A) cancelled
    02:14:16 03:33:26 79 provider-turn.start (thread E) cancelled
    03:00:42 03:37:38 37 provider-turn.start (thread A) succeeded
    • The server log is silent from 00:41 to 03:37 except for one environment api request failed with errorTag: 'Interrupt' at 00:40:59. At 03:37:32 everything drains at once: dozens of queued thread.auto-settle attempts fail with "changed before automatic settlement".
    • The Claude process of thread A received the user message only at 03:37:38, as a "Provider context handoff". It answered 4 s later. So the provider was idle the whole time, and the turn start was waiting on a worker.
    • No OOM, no restart, and no stuck git/glab children afterwards. It cleared by itself after about 3 h, unlike [Bug]: A child that ignores SIGTERM never lets ProcessRunner's timeout return, leaking a VcsProcess permit; after 8 leaks every git call and every new turn hangs forever #20419.
    • Over three days, 433 of 439 provider-turn.start effects finished in under 15 s; the other 6 are the rows above. So a deadline on the effect itself, not only on the lease, would separate these cleanly.

    The trigger seems to be a run cancelled or interrupted right before (thread A run cancelled at 00:27, steer/interrupts from an automation client). After that, the next start on the same thread never resolved until a worker freed up.

    Workaround we run meanwhile: a watcher that reads the outbox every 15 s. It alerts on any effect pending or running for more than 15 s (checkpoint: 60 s). After 30 s more it kills the Claude CLI process of that thread, matched via --session-id in provider_threads, so the start fails and frees its worker.

  4. josephv123 commented on Oct 18, 2026

    @josephv123
    Contributor

    Note

    Claude Opus 5.5 in Claude Code, writing on behalf of @josephv123.

    Another occurrence, Claude only, on nightly 0.1.1-nightly.20261018.3315 (Linux). Here the blocking call is identified.

    Twelve Claude threads had stalled while their agents slice was throttled at MemoryHigh (the claude processes were alive but barely running). Stop was sent to all 12 at 13:41:22. From the trace and orchestration_v2_effect_outbox:

    • All 12 provider-turn.interrupt effects were queued within about a second.
    • Only four started. They took 140.5, 220.3, 132.9 and 209.9 s.
    • Each of the other eight started 20–43 ms after an earlier one finished, so never more than four ran at once. All twelve took 3 min 43 s.
    • 24 preview.cleanup / terminal.cleanup effects for those threads waited 217–219 s behind them. They normally take about 0 s.

    The time was spent in ClaudeAdapterV2.interruptTurn awaiting existing.query.interrupt, the SDK's interrupt control request. That call has no deadline: the SDK resolves it only when the CLI sends its control_response. The existing 10 s bound covers only the close that follows. Each of the four spans ended exactly 2.0 s, the SDK's close grace, after its CLI finally answered. Those CLIs had been silent for up to 369 s.

    #21303 bounds that one call and always continues to the close, so Stop ends the turn as interrupted. It combines #16419 and #16439 on current main. In a live check with a SIGSTOPped real Claude CLI, the interruptTurn span went from 90.7 s (it only ended after SIGCONT) to 12.0 s, and the frozen CLI was killed 17 s after Stop. It doesn't touch the worker pool itself, so it fixes one source of this stall, not the issue.

  5. added
    needs more infoInitial triage showed no bug. Awaiting more info
    and removed
    needs more infoInitial triage showed no bug. Awaiting more info
    on Oct 22, 2026
  6. josephv123 commented on Oct 22, 2026

    @josephv123
    Contributor

    Note

    Claude Opus 5.5 in Claude Code, writing on behalf of @josephv123.

    Reproduced on the current nightly, 0.1.2-nightly.20261021.3499 on Linux (t3 serve, default four effect workers), with the outbox snapshot and server log taken while it was stuck: secret gist (outbox-snapshot.txt, server-log.txt, plus the trace spans and the stub).

    Setup: a Codex instance whose app-server stays alive but never answers thread/start (a small proxy in front of codex-cli 0.163.0 that drops it, codex-wedged.js in the gist), four Codex threads, then one Claude thread, on a separate server with a throwaway home.

    • Pending Claude provider-turn.start: queued at 13:29:27.885Z, attempt count 0.
    • The four running effects, all provider-turn.start on Codex threads, claimed at 13:29:20.093, 13:29:21.014, 13:29:21.912 and 13:29:22.803Z. At the snapshot (13:39:56Z) they were 636, 636, 635 and 634 s old, about 605 s past their 30 s leases, with attempt count 1.
    • The Claude start was pending for 629 s at the same moment.

    Each of the four was blocked in a single CodexAppServerProtocol.request (thread/start under ProviderTurnStartService.loadFromProvider). Each span ran about 639 s and ended only when I killed the app-server at 13:39:59.533Z. Nothing timed out or reclaimed the expired leases in the meantime. The Claude start was claimed 18 ms later (13:39:59.551Z), 631.7 s after it was queued, and its turn then completed normally (checkpoint captured at 13:40:03Z).

  7. nadam54321 commented on Oct 24, 2026

    @nadam54321

    Same here on T3 0.1.1 (t3 serve) with Codex 0.162.1, in a Linux VM with 4 GB and no swap. From about 15:30 UTC on 23 October four provider-turn.start effects went to running and stayed there; their leases expired and were never reclaimed, so 55 turn starts and 53 title generations waited behind them, and no turn finished on any thread until we stopped the runs and restarted T3 at 21:52 UTC.

    The trigger was the agents slice holding at MemoryHigh (2.0 GB of 4 GB): 3.6 million high events, 96% of its time stalled, Codex's threads in __mem_cgroup_handle_over_high, while the VM had 846 MB free. In the slice were the app-server with about 38 loaded threads, most waiting on command approvals, and 31 processes of one MCP plugin, about 78 MB each.

    Adding swap (Ubuntu's zram) has kept the slice from stalling since, even with 30 turns waiting on approvals at once. It only removes this trigger: the four workers never freed themselves, so any provider that stops answering can still stall every thread.

  8. seankoji commented on Oct 27, 2026

    @seankoji

    Additional live Linux incident evidence from 28 October 2026 (AEDT, UTC+11).

    A read-only pre-recovery inspection of the backend found all four default effect-worker slots occupied by provider-turn.start. Their associated threads were configured for Codex. OpenCode threads and later control/cleanup work were also stuck.

    Running start Attempt count Lease expiry (UTC)
    A 1 2026-10-27T00:13:32.080Z
    B 1 2026-10-27T07:43:26.385Z
    C 1 2026-10-27T21:47:53.359Z
    D 1 2026-10-27T23:24:38.935Z

    All four leases had expired and their owner matched the backend PID in runtime metadata. In the same inspection window, the outbox held 29 pending provider-turn.interrupt, 82 provider-session.detach, 12 further starts, 30 delegated stops and 4 checkpoints. The installed server bundle sets DEFAULT_EFFECT_WORKER_CONCURRENCY = 4.

    The saved incident sequence was stalled existing threads, acknowledged Stop requests that did not clear them, and later threads accepted without provider progress. Reconnecting the desktop did not clear it. A targeted Stop later interrupted one parent, but another queued Codex start immediately occupied the freed slot and stalled. That makes clearing one parent insufficient to clear the shared backlog. Saved histories and Git work were preserved; recovery is being handled separately.

    Environment and limits:

    • Linux Mint standalone server, macOS desktop client, Codex and OpenCode configured.
    • This is a local terminal-fix candidate based on 0.1.3-nightly.20261026.3726; its activation receipt records source e900e620238baa804cf487efb21fe659ca9a189b. Runtime metadata separately reports serverVersion: 0.1.2. I have not reproduced on an unmodified current build.
    • This was observation of a live incident, without fault injection. The exact native call holding each start is still unknown.
    • fix(server): release workers when providers stop answering #23854 is open and has not been tested or installed here. The merged fix(server): bound hung delegated stop effects so later runs can start #25675 addresses a hung delegated stop; the running effects in this snapshot are starts.

    This appears to be another occurrence of #16636. The initiating provider failure and a permanent fix remain unverified. A separate, approved same-runtime restart is being attempted for recovery; this comment does not claim its outcome.

    Reported through Codex, GPT-6.1 Sol at xhigh effort.

  9. piotrnowakowski commented on Oct 29, 2026

    @piotrnowakowski

    Note

    Claude Opus 5.5 in T3 Code, writing on behalf of @piotrnowakowski. Diagnosed from statev2.sqlite and the provider/server logs on their machine.

    Three more occurrences on Windows 11, T3 Code desktop 0.1.3, shared Codex app-server codex-cli 0.165.0, Claude Agent as the second provider. All three were triggered by an orchestrator fanning out Codex delegates via delegate_task in waves of five.

    Every time, exactly four Codex provider-turn.start effects were claimed (attempt_count = 1) and never finished. Everything after them stayed pending with attempt_count = 0, on every provider, until the app was restarted and the restart cancelled them.

    Incident (UTC) Running forever (attempt 1) Waiting behind them Stall Recovery
    2026-10-28 22:41 4 × Codex provider-turn.start 4 starts + 5 more that ran ~34 min late, 8 checkpoint.capture, 1 thread-title.generate ~35 min restart, 23:16
    2026-10-28 23:26 4 × Codex provider-turn.start 17 starts, 1 steer, 3 checkpoints, 1 detach, 1 title ~19 min restart, 23:44
    2026-10-29 22:24 4 × Codex provider-turn.start 6 starts (5 of them Claude threads), 1 provider-session.detach ~31 min restart, 22:55

    For comparison, provider-turn.start on the same day before the stall: 286 effects, mean 4.1 s, max 48 s.

    From the user's side, a Claude thread showed the sent message plus the "thinking" indicator for 30 minutes. Its provider-turn.start was never claimed, and its provider log has nothing between the previous turn and the restart.

    What the Codex app-server did (29 Oct, shared session log)

    The app-server was not fully wedged. Only the four new turns hung:

    22:24:35-36  id 346-349  thread/start   -> answered within ~1 s
    22:24:43     id 350      turn/start  (thread ...d85e)  -> never answered
    22:24:45     id 351      turn/start  (thread ...d9cb)  -> never answered
    22:24:46     id 352      turn/start  (thread ...d8d3)  -> never answered
    22:24:46     id 353      turn/start  (thread ...d9f3)  -> never answered
    22:33-22:53  id 354-379  thread/unsubscribe          -> each answered in 1-3 ms
    

    So T3 kept talking to the app-server normally. The four workers stayed blocked on four turn/start calls that never returned, with no deadline.

    Each new Codex thread starts about 15 MCP servers from ~/.codex/config.toml and plugins. Six of them are local stdio: playwright, node_repl, cua_repl, blender, mlflow and openai-api-key-local-confirmation. In the stalled wave, several were still failing or timing out while turn/start was pending: mlflow "timed out awaiting tools/list after 60s" (final status at 22:26:13), playwright "timed out awaiting tools/list after 30s", plus auth failures for two remote ones. Every MCP server had reached ready or failed by 22:26:13, but the four turn/start calls still never returned. This may be related to openai/codex#42740 (Windows stdio MCP servers exhausting the blocking pool), but I could not confirm it from the logs. Unlike that issue, the app-server here kept answering other requests.

    Either way, the T3 side matches this issue: one provider's hung turn/start blocks all four shared effect workers, and that stalls Claude threads too. A per-effect deadline on provider-turn.start, or per-provider workers, as proposed in #23854 / #21321, would have contained all three incidents.

    Workaround on our side for now: disable rarely used stdio MCP servers in ~/.codex/config.toml (enabled = false), and restart T3 when every thread stalls.

  10. ethanjnrg commented on Oct 30, 2026

    @ethanjnrg

    Additional observed occurrence on macOS arm64 (27.0.1), T3 Code Beta 0.1.5, on 2026-10-30. Diagnosed read-only from the local run projection, effect outbox, server trace and installed application bundle. The currently installed Codex CLI reports 0.166.0; I have not independently established the exact executable/version of the earlier stalled app-server.

    This corroborates the unbounded provider-control request portion of this issue. I did not establish exhaustion of all four workers or a cross-provider outage.

    Observed sequence (UTC)

    • 16:56:06.783: preceding run completed normally. Its background business command had already finished, and its owned browser/process exited.
    • 17:35:28.978: answering an earlier asynchronous user-input question created the next run in the same existing Codex conversation.
    • 17:35:29.027660834: CodexAppServerProtocol.request began under CodexAdapterV2.resumeThread.
    • 22:57:43.573: the user pressed Stop.
    • 22:57:43.620793917: the protocol request ended as Interrupted, after 19,334,593.133 ms (5h 22m 14.593s).

    The UI appeared to be working throughout, but the run projection has startedAt: null and the final activity says Run interrupted before provider start. This was a stalled startup operation, not five hours of model execution.

    The matching orchestration_v2_effect_outbox row records:

    effect_type:   provider-turn.start
    attempt_count: 1
    created_at:    2026-10-30T17:35:28.985Z
    completed_at:  2026-10-30T22:57:43.578Z
    status:        cancelled
    last_error:    Run <redacted> was interrupted before its provider turn started.
    

    The nested ProviderTurnStartService.start span lasted 19,334,600.559 ms, also ending as interrupted. No automatic timeout/failure/retry settled this attempt during that interval. Later user messages started and completed normally; this report does not claim a restart was necessary or identify what restored the transport.

    Installed-source check

    In the shipped apps/server/dist/binCli-1k0PQRR7.mjs:

    • Lines 235641–235655: CodexAdapterV2.resumeThread calls client.raw.request("thread/resume", { threadId, excludeTurns: true, ... }) and awaits it without a local timeout.
    • Lines 230239–230251: CodexAppServerProtocol.request registers a pending deferred, offers the outgoing request, then awaits the deferred with interruption cleanup, but no request deadline.
    • Line 304498 still sets DEFAULT_EFFECT_WORKER_CONCURRENCY = 4.

    The fact that excludeTurns: true is already present distinguishes this from the older full-history resume response problem in #6399.

    Scope / uncertainty

    This is one live observation, not a deterministic reproduction. I have not reproduced it with fault injection. The exact reason the resume request did not complete is unknown: these records do not prove an app-server exit, stdio backlog, OOM, provider usage limit, or four-worker saturation. #24300 is related to unanswered shared app-server control requests, but its backlog diagnosis has not been established here.

    The confirmed containment gap is that a provider-control RPC remained pending for over five hours before manual cancellation, with neither a visible startup error nor bounded recovery. A deadline on the startup/control operation would contain this case even if the initiating transport/provider failure remains unknown.

    Reported by Codex (GPT-6.1 Sol) at the user's explicit request. Conversation content, business data, local identifiers and credentials are omitted.

  11. esukram commented on Oct 31, 2026

    @esukram

    Note

    AI-generated report. Written by Claude Opus 5.5 (Claude Code, run through t3 triage) on behalf of @esukram. Diagnosed read-only from statev2.sqlite, the server trace, the provider event logs and Codex's own logs_2.sqlite on their machine. A human reviewed it before posting.

    Another occurrence on 0.1.6-nightly.20261030.3940 (Linux, t3 serve as a systemd user service, codex-cli 0.166.0 as the shared app-server, Claude Agent as the orchestrating provider). It adds two things to this issue: the trigger that wedged Codex, and a second blocking path, the per-worktree workspace lease, that stalls Claude runs before the worker pool is exhausted.

    Trigger: Codex drops its stdio client on EAGAIN but keeps running

    Codex's own log (~/.codex/logs_2.sqlite) at the moment it went silent:

    2026-10-31 17:29:04 ERROR codex_app_server_transport::transport::stdio  Failed to write to stdout: Resource temporarily unavailable (os error 11)
    2026-10-31 17:29:04 WARN  codex_app_server::transport  dropping message for disconnected connection: ConnectionId(0)
    ... 307 more "dropping message for disconnected connection" lines, through 18:09:49
    
    • The app-server's stdout is a non-blocking socket: /proc/<pid>/fdinfo/1 shows flags: 04002, i.e. O_RDWR|O_NONBLOCK.
    • When one write hit EAGAIN, Codex marked connection 0 as disconnected. It kept reading stdin but answered nothing.
    • The process stayed alive, idle in futex_do_wait, so T3 never noticed.
    • The T3 provider log shows outgoing thread/start (ids 10, 15, 16), thread/unsubscribe (ids 11–13) and turn/interrupt (id 14) from 17:45 to 18:09. Nothing came back.
    • On the Codex side this resembles codex exec --json panics with "failed printing to stdout: Resource temporarily unavailable (os error 11)" when the stdout reader is slow openai/codex#55242 (codex exec --json panics on the same EAGAIN when the stdout reader is slow).
    • On the T3 side, a liveness check or request deadline on the shared Codex session would have caught it.

    Second blocking path: the workspace lease

    ProviderTurnStartService.start wraps startup in withWorkspaceLease(worktreePath, …). That is a per-path Semaphore(1), shared with git workflow ops, terminal open and checkpoint rollback.

    A delegate_task child inherits the parent's worktree. So a Codex delegate whose thread/start never returns holds the parent's worktree lease indefinitely. In this case:

    UTC Effect Thread Outcome
    17:45:28 provider-turn.start Codex delegate of parent A (worktree W1) hung on thread/start id 10, holding W1's lease
    17:47:08 provider-turn.start parent A itself (Claude), a pr-watch wake blocked on W1's lease; run stayed starting, three more runs queued behind it
    17:59:31 provider-turn.start Codex delegate of parent B (worktree W2) hung on id 15 (later interrupted and cancelled)
    18:09:46 provider-turn.start another Codex delegate of parent B hung on id 16
    • At 18:10, 3 of 4 effect workers were held, and leases showed lease_expires_at NULL.
    • The Claude parent on W1 was already dead because of the lease, before the pool ran out.
    • The trace backs this up: no ProviderTurnStartService.start child span (not even prepareRetryHandoff) completed for the blocked Claude start.

    Recovery

    systemctl --user stop on the t3code-codex-*.scope, with no T3 backend restart:

    • All three hung starts settled within about 2 s (18:14:21–23).
    • T3 spawned a fresh app-server.
    • The delegates and the blocked Claude run started normally.

    Suggestions (beyond the deadline and per-provider workers already proposed here)

    1. Treat a shared provider session that answers no request within N seconds as dead, and recycle it. Here that would have self-healed.
    2. Don't hold the workspace lease across an unbounded provider RPC. Or bound the leased section, so that a delegate can't block its parent's own turns in the same worktree.
    3. Possibly avoid handing Codex a non-blocking stdout, or report the Codex-side EAGAIN handling upstream.

    Environment

    • T3 Code server 0.1.6-nightly.20261030.3940 (triage CLI 0.1.6), t3 serve under t3code.service; client is the web UI against the LAN server
    • Arch Linux, kernel 7.2.11, x64, Node v26.12.1
    • codex-cli 0.166.0 (shared app-server), Claude Agent provider with claude-opus-5-5

    Model: Claude Opus 5.5. Harness: Claude Code. Orchestrator: T3 Code (t3 triage).

  12. esukram commented on Oct 31, 2026

    @esukram

    Note

    AI-generated follow-up. Written by Claude Opus 5.5 (Claude Code, run through t3 triage) on behalf of @esukram. It follows up on the comment above. Evidence comes from host monitoring (Dynatrace OneAgent, plus T3's own OTLP metrics), a cgroup watcher, the local trace and Codex's logs_2.sqlite. A human reviewed it before posting.

    Same machine, same build (0.1.6-nightly.20261030.3940, codex-cli 0.166.0). The wedge recurred twice more after recycling the Codex app-server: three times in ~80 minutes (17:29, 18:30, 18:51 UTC). Each time the cure was the same: stop the t3code-codex-*.scope. Every occurrence was a Codex delegate_task child of the same Claude orchestrator thread, sharing its worktree.

    Ruled out

    Hypothesis Evidence Verdict
    T3 event loop blocked, so stdout not drained t3_server_event_loop_delay_max (T3's own OTLP metric, 1-min max) was 15 ms at 17:29 and 13 ms at 18:30, never above 38 ms from 17:20 to 18:42 ruled out
    Host starvation Host CPU 5–14 %, load15 0.4–0.7, memory ~40 % at all three times ruled out
    cgroup memory or socket throttling (MemoryHigh on app-t3code-agents.slice) 10 s watcher of memory.events: high and sock_throttled flat for >5 min before the 18:51 EAGAIN, memory PSI some avg10=0.00, slice at 7.7 of 9.4 GiB ruled out

    The 18:51 occurrence logged the same Failed to write to stdout: Resource temporarily unavailable (os error 11) as the first. The process stayed alive and dropped every later message.

    Likely mechanism: T3 stops reading Codex's stdout while waiting on its own lock

    The reader in packages/effect-codex-app-server/src/protocol.ts (~L406–430) processes stdout one line at a time and handles each line before pulling the next chunk.

    • Notification handlers run inline in that loop: handleNotification → options.onNotification → client.ts dispatchNotification → the adapter handlers.
    • In CodexAdapterV2.ts, the item/started, item/completed, thread/closed and thread/goal/* handlers all run under turnTerminalizationPermit.withPermits(1) (e.g. ~L4528, L4694, L4835, L5157). That is one Semaphore(1) per adapter session, and with the shared app-server that means one for all Codex threads.
    • So any holder of that permit that waits on something pauses the stdout reader for every Codex thread. No event-loop delay and no CPU are needed.
    • During a fast item/agentMessage/delta stream, Codex then fills its (non-blocking) stdout socket, gets EAGAIN, and treats the client as gone.

    This matches every measurement above: idle host, healthy event loop, and EAGAIN mid-stream. I have not identified which permit holder waits.

    Timing correlation

    • At 18:30:55 and 18:51:26 the orchestrating Claude parent's run completed in the same second Codex went silent.
    • At 18:30 the drop came first: first dropping message at 18:30:55.494, RunFinalizationService.finalize at 18:30:55.868. At 18:51, finalize (18:51:26.978) falls inside the 10 s window of the EAGAIN.
    • So the two are related but the order is not causal. It may be worth checking what a parent's turn end does to its delegates' Codex state, such as delegated-completion delivery.

    Version history on this machine

    Codex T3 nightly Result
    0.166.0 (since Oct 30, 02:03) 0.1.6-nightly.20261029.3892 full day, no wedge (lighter Codex use)
    0.166.0 0.1.6-nightly.20261030.3940 (Oct 31, 02:09) three wedges

    3892...3940 has 71 commits. None touches the protocol reader, but CodexAdapterV2.ts gained the generated-image state (#19066, +119 lines). Load also differed, so this is not proof of a regression.

    Suggestions

    1. Don't run adapter notification handlers inline in the stdout read loop. Hand them off through a queue, so that a waiting handler can't apply backpressure to the Codex process.
    2. Or, at minimum, never hold turnTerminalizationPermit across an RPC or I/O wait.
    3. As noted before, detect a shared app-server that has stopped answering (request deadline plus recycle) instead of waiting forever.

    Side note: the server's OTLP trace and metric export to the collector stopped at ~18:43–18:45 UTC, while local server.trace.ndjson kept writing. Possibly unrelated.


    Model: Claude Opus 5.5. Harness: Claude Code. Orchestrator: T3 Code (t3 triage).

  13. esukram commented on Nov 1, 2026

    @esukram

    Note

    AI-generated update. Written by Claude Opus 5.5 (Claude Code, run through t3 triage) on behalf of @esukram. A human reviewed it before posting.

    This still happens on 0.1.7-nightly.20261031.4004 with codex-cli 0.166.0. On Nov 1 at 13:25:29 UTC the shared Codex app-server logged os error 11 on stdout, then went silent. Two Codex delegate_task starts (thread/start ids 27 and 32) were never answered, holding 2 of the 4 workers until we stopped the t3code-codex-*.scope by hand. So it isn't specific to 20261030.3940.

    The likely cause on T3's side, Codex notification handlers stalling the stdout reader, is now tracked separately in #28697.


    Model: Claude Opus 5.5. Harness: Claude Code. Orchestrator: T3 Code (t3 triage).

  14. dellintosh commented on Nov 4, 2026

    @dellintosh

    Note

    AI-generated follow-up. Written by Claude Opus 5.5 (Claude Code) on behalf of @dellintosh, from the server trace, the service log, statev2.sqlite (read-only) and Codex's logs_2.sqlite. A human reviewed it before posting.

    Same wedge on the stable v0.1.6 release, on Linux, headless. Adding it because the setup differs from the Windows reports here, and the effect outbox shows the four-worker hold directly.

    Environment: Ubuntu 26.04.1 LTS (kernel 7.0.0-38), t3 serve as a systemd user service, t3 v0.1.6, codex-cli 0.166.0 as the shared app-server, Claude Code 2.1.318 as a second provider. About ten threads across two projects, several of them Codex delegate_task children.

    What happened (UTC, 2026-11-04):

    • 02:50 to 03:02: four provider-turn.start effects were claimed (attempt_count 1). At least two were Codex delegate_task child starts.
    • 02:57:37 and 02:57:39: Codex's log records thread/resume (request 253) and thread/start (request 254) arriving. Its last line of any kind is at 02:59:10, then nothing until the restart at 10:44. No error and no exit message.
    • 03:27:37: the service log shows four orchestration V2 provider event ingestion failed warnings (RunExecutionIngestError, caused by ProviderAdapterEventStreamError on provider-session:provider-instance:codex:shared, caused by Provider session released: idle_timeout), then orchestration-v2.provider-session-scope-close-timeout 30 seconds later.
    • For the next 7 h 17 min no turn started on any provider. Claude threads sat in starting too. Effects created later (another Codex start, a title generation, a delegated-task stop) stayed at attempt_count 0.
    • At 10:42 the t3code-codex-*.scope no longer held a codex process (only an unrelated dev server an agent had started), yet the four starts still held their workers. So the hold outlives the provider process, not only a live but silent one.

    Recovery: t3 service restart. The four held starts and the queued one ended cancelled ("Cancelled because the server shut down before the provider work completed."), and the queued title and stop effects ran within seconds. Claude threads resumed by themselves; Codex threads came back cancelled and needed their last message resent.

    Outbox rows (read-only query):

    effect_type            status     attempts  created_at (UTC)  completed_at (UTC)
    provider-turn.start    cancelled  1         02:50:07.910      10:44:27.143
    provider-turn.start    cancelled  1         02:56:23.822      10:44:27.133
    provider-turn.start    cancelled  1         02:57:37.500      10:44:27.200
    provider-turn.start    cancelled  1         03:02:10.965      10:44:27.502
    thread-title.generate  succeeded  1         04:32:45.920      10:44:40.318
    delegated-tasks.stop   succeeded  1         04:36:36.258      10:44:35.101
    provider-turn.start    cancelled  0         04:36:59.221      10:44:27.307
    

    I checked the v0.1.7 release notes and found nothing aimed at this. The open PRs #23854, #21321 and #28711 look like the fixes.

    Model: Claude Opus 5.5 via Claude Code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething is broken or behaving incorrectly.via-triageFiled through npx t3 triage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions