Skip to content

[Bug]: Restart continuation skips all Codex threads when shutdown preparation does not finish (shared session stopped by first reconciled thread) #15749

New issue
New issue
Closed
Closed
[Bug]: Restart continuation skips all Codex threads when shutdown preparation does not finish (shared session stopped by first reconciled thread)#15749
Labels
bugSomething is broken or behaving incorrectly.via-triageFiled through npx t3 triage

Description

@Dropje97
Dropje97
opened on Oct 4, 2026

Before submitting

  • I searched existing issues and did not find a duplicate.
  • I included enough detail to reproduce or investigate the problem.

Area

apps/server

Steps to reproduce

  1. Enable "Continue threads after server update" (continueThreadsAfterServerUpdate: true, global, no project overrides).
  2. Have several threads running at the same time, at least one on Codex and at least one on Claude. All Codex threads are bound to the single shared session provider-session:provider-instance:codex:shared.
  3. Restart the service (systemctl --user restart t3code.service) in a way where prepareForShutdown does not record any continuations before the launcher force-kills the server. In our case the server was under load (event loop stalls of 2–20 s), and the launcher's fixed TERMINATE_GRACE_MS = 5_000 in apps/server/src/serviceLauncher.ts SIGKILLed it about 5.4 s after Stopping.
  4. Wait for startup reconciliation.

Expected behavior

Every root run that was running when the server went down gets a restart-continuation ("Continue where you left off."), regardless of provider.

Actual behavior

Only Claude threads were continued. Every Codex run was cancelled by startup reconciliation and never got a continuation. That covered one top-level thread and five delegated tasks across two projects.

Cause, from reading ProviderRuntimeRecoveryService.reconcile("startup") and restartContinuationRun:

  • Reconciliation runs per thread, sequentially. For each thread, reconcileProjection emits provider-session.updated → stopped for every bound session that is not stopped yet.
  • Claude has one session per thread, so each Claude thread still sees its own session as ready when restartContinuationRun evaluates it.
  • Codex threads all share provider-session:provider-instance:codex:shared. The first reconciled thread bound to it, an old idle thread with no active run, marked the shared session stopped. Every Codex thread reconciled after it then failed this check, so it got no continuation: 
    (liveTurnRequired && (session.status === "stopped" || session.status === "error"))

The shutdown path (prepareForShutdown) does not have this problem, because it runs while sessions are still live. In an earlier restart the same day it finished in time (continuation effects written ~4.5–4.8 s after Stopping), and six Codex runs were continued correctly. The startup fallback is the path that skips Codex, and it is the only path left whenever shutdown preparation does not finish within the 5 s grace.

From reading the code, both points still apply on main @ 4ee6bfd: the reconcile loop still stops bound sessions per thread, restartContinuationRun still rejects a stopped session, and TERMINATE_GRACE_MS is still 5_000. main now calls recoverDelegatedTask for declined continuations, which should at least report back to the parent. I did not reproduce this on main.

Possible directions:

  • Evaluate continuation eligibility for all threads before emitting any session stops. Alternatively, do not stop a shared session in a per-thread reconcile while other bound threads have running runs.
  • Make the launcher's terminate grace configurable or longer, or let prepareForShutdown run before the grace timer starts.

Impact

Major degradation or frequent failure

Version or commit

0.0.46-preview.20261002.2598 (source b37fcea); code paths also checked on main @ 4ee6bfd

Environment

Linux (Debian LXC), official @t3code/t3-linux-x64 executable under systemd (KillMode=mixed, TimeoutStopSec=90s), Codex app-server provider (shared session) + Claude Agent provider

Logs or stack traces

# systemd journal, local time (UTC+2): launcher force-kills after ~5.4 s
22:03:13.195 systemd: Stopping t3code.service - T3 Code server...
22:03:18.656 systemd: t3code.service: Killing process ... (codex) with signal SIGKILL.
# no effect:restart-continuation rows with command:restart-prepare:* for this restart

# provider-session.updated events during startup reconcile (UTC)
20:03:30.891 provider-session:provider-instance:codex:shared  stopped  command:runtime-reconcile:startup:<idle codex thread>
20:03:33-39  provider-session:provider-instance:claudeAgent:thread:<id>:<id>  stopped  (one per Claude thread)

# resulting restart-continuation effects: 5, all claudeAgent
# runs cancelled by this restart with no continuation: 6 codex

Workaround

Manually send a follow-up message to each interrupted Codex thread, or have the parent re-delegate the task.

Activity

  1. juliusmarminge commented on Oct 4, 2026

    @juliusmarminge
    juliusmarminge
    on Oct 4, 2026
    Member

    Note

    Grok responding on behalf of Julius.

    Triage

    Thanks for the really thorough investigation, @Dropje97! Your journal and event timeline lined up exactly with the code.

    What I found

    Confirmed on main @ 4ee6bfd50ef4.

    • Codex shares one session across threads. Codex and OpenCode 2 use a single session per provider instance: provider-session:provider-instance:<instance>:shared.
      • The ID comes from IdAllocator.derive.providerSession, which is used when supportsMultipleProviderThreadsPerSession is true.
      • That row is shared, so a provider-session.updated from any bound thread sets status for every thread (ProjectionStore, ON CONFLICT(provider_session_id)).
    • Startup reconcile stops it on the first thread. ProviderRuntimeRecoveryService.reconcile("startup") walks recovery threads oldest-first, and that includes idle threads still bound to a live session.
      • For each thread, reconcileProjection commits provider-session.updated → stopped for every live session it sees.
      • restartContinuationRun rejects a live turn once that session is stopped or error.
      • So an older idle Codex thread can stop the shared session before any running Codex thread is evaluated. The running threads are then cancelled with no continuation. Claude isn't affected because each thread has its own session.
    • Shutdown doesn't hit this, but it can run out of time. prepareForShutdown records continuations while sessions are still live, so it avoids the problem. But terminateChild in apps/server/src/serviceLauncher.ts sends SIGKILL TERMINATE_GRACE_MS (5000 ms) after SIGTERM, and systemd's TimeoutStopSec doesn't extend that. A stalled event loop can miss the window, which leaves only the startup path.
    • recoverDelegatedTask doesn't cover these runs. It only runs after a provider-runtime.continue effect was written and then declined. The skipped Codex runs never get that effect.

    Likely fix area

    • Option 1: decide eligibility first. In ProviderRuntimeRecoveryService, evaluate continuation eligibility for every bound thread before any thread commits a stop on a shared session.
    • Option 2: protect shared sessions. Don't stop a shared session in a per-thread reconcile while other bound threads still have running runs.
    • Separately: the launcher grace. Revisit TERMINATE_GRACE_MS (longer, configurable, or started after prepareForShutdown completes). This would make the startup fallback rarer but wouldn't fix it on its own.

    A maintainer will decide on the fix direction.

  2. juliusmarminge
    added
    bugSomething is broken or behaving incorrectly.
    via-triageFiled through npx t3 triage
    on Oct 4, 2026

  3. waffleflopper commented on Oct 7, 2026

    @waffleflopper
    waffleflopper
    on Oct 7, 2026

    Another occurrence, on a different install: the Linux desktop app restarted by its in-app updater, not systemd.

    What happened: A Codex (gpt-6.1-sol) thread was mid-turn, waiting on a native Codex subagent, when the desktop updater restarted the app. "Continue threads after restarts" is on. After the restart the thread was settled, the subagent was cancelled, and nothing resumed it. I had to send "continue where we left off" by hand.

    Evidence (UTC, 2026-10-07):

    • Run 57 was running: its attempt, root node and provider turn were all still open and were cancelled by startup reconcile at 04:56:37.147Z. Its native subagent (/root/personnel_native) was cancelled at the same moment.
    • The old backend's last trace span is at 04:56:30.68Z. There is no command:restart-prepare:* effect for this restart, so prepareForShutdown didn't finish on the desktop either.
    • Startup reconcile first processed an idle Codex thread (last run completed 2026-10-05) bound to provider-session:provider-instance:codex:shared, at 04:56:36.740Z. The running thread was processed at 04:56:37.147Z, and no provider-runtime.continue effect was written for it.
    • Earlier restarts on 2026-10-04 and 2026-10-06 on the same install continued Codex threads correctly (restart-prepare and startup effects both present), which fits the order-dependent failure described here.

    Environment: T3 Code desktop 0.0.46-nightly.20261006.2752 (d8d037e), Linux x64 (Arch / Omarchy, kernel 7.2.3), installed from GitHub releases, nightly channel, Codex provider with a shared session.

    So the startup fallback hits desktop updater restarts too, not just systemd's 5 s grace. #15820 looks like it covers the startup half.

    Investigated and drafted with Claude Opus 5.5 (claude-opus-5-5) in Claude Code via t3 triage.

  4. moritzwesemann commented on Oct 9, 2026

    @moritzwesemann
    moritzwesemann
    on Oct 9, 2026

    Also seeing this on macOS 27.0.1 (arm64), desktop nightly 0.0.46-nightly.20261009.2873, using Codex / gpt-6.1-sol.

    During an in-app update on October 9, three active Codex threads stopped despite continueThreadsAfterServerUpdate: true and no project override. All resumed after I manually sent “continue.”

    Local traces and read-only database checks show:

    • Startup recovery cancelled all three runs at 15:46:27.137–15:46:27.245Z.
    • No restart-continuation effects or automatic continuation messages were recorded for them.
    • The recovery summary reported terminalizedRuns: 3, stoppedSessions: 2, requeuedEffects: 0.

    This matches the shared-session failure described here. I haven’t tested #15820.

    Investigated with Codex (gpt-6.1-sol).

  5. t3dotgg commented on Oct 21, 2026

    @t3dotgg
    t3dotgg
    on Oct 21, 2026
    Member

    Note

    🤖 Claude Opus 5.5 responding on behalf of Theo

    Thanks for the report. This is fixed on main in #19750. Please test on the nightly built from main on 2026-10-21, or on the next stable release. If the problem still occurs there, please open a new issue that links to this one, with your version and steps to reproduce it.

  6. t3dotgg
    closed this as completedon Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething is broken or behaving incorrectly.via-triageFiled through npx t3 triage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions