Skip to content

[Feature]: Pause, update, and resume active threads as one operation #15186

Closed

Description

@kossoy

Area

apps/server

Requested workflow

Please provide one operation to pause active threads → update the server → resume those threads. With several threads running, updating currently requires deciding when to interrupt them and then potentially reopening and prompting each thread individually.

The operation should capture the pre-update active-thread set, let work pause at a safe boundary or explicitly interrupt it, preserve the recovery information across the update, and resume only that captured set. Show which threads resumed and allow retrying the ones that did not. Threads already stopped by the user should stay stopped.

The existing Continue threads after restarts preference is useful, but does not expose this coordinated maintenance workflow. The current V1 → V2 upgrade also appears to have a migration-specific limitation described below.

Versions and environment

  • Running server: 0.0.45-nightly.20261002.2561, orchestration protocol 1.
  • Staged replacement: 0.0.46-nightly.20261003.2623, orchestration protocol 2.
  • Linux remote server managed by a custom systemd user service; macOS desktop client.

Example scenario

  1. Run the protocol-1 server with multiple active threads.
  2. Enable Settings → General → Continue threads after restarts, including any relevant project overrides.
  3. Update/restart into 0.0.46-nightly.20261003.2623 while those threads are still running.
  4. Resume the pre-update active set together, without manually visiting each thread.

Expected behavior

An update should preserve the active-thread set and honor the continuation preference across the migration. If native provider sessions cannot be retained, offer a single action to continue those threads from their imported history and clearly explain that limitation.

Migration limitation motivating this request

This part is source-derived, not a completed runtime reproduction. I inspected both shipped nightly binaries before restarting the active server; I have not performed this upgrade with active threads.

The staged binary contains these source regions:

  • apps/server/src/orchestration-v2/legacy/LegacyV1ThreadImporter.ts: importedThread() sets activeProviderThreadId: null and historyOrigin: "v1_import". Imported messages and turn items have runId: null and no provider thread/turn identity. Streaming legacy items become interrupted. The importer writes thread metadata and transcript events, but does not import runs or provider sessions.
  • apps/server/src/orchestration-v2/RestartContinuation.ts: restartContinuationRun() requires a recoverable V2 run and a strong native provider-thread reference. continueRestartedRun() uses continueThreadsAfterServerUpdate to dispatch a continuation for a qualifying source run.
  • The old server's serverRuntimeStartup.ts supports restart continuation through the V1 session binding and resume cursor. The new binary does not contain the old continueAfterServerUpdate marker.

This leaves no apparent path from an active V1 session to a qualifying V2 restart continuation. Please confirm whether another migration path covers this case; I am not claiming a reproduced runtime bug.

Impact and workaround

This blocks a convenient update while multiple threads are active. Waiting for all work to finish avoids interruption; manually continuing each thread after migration is the obvious fallback. Neither provides a bulk pause/update/resume workflow. No workaround was tested as part of this report.

The proposed workflow would also cover this boundary by carrying enough interrupted-run/resume state across migration, or keeping a recoverable pre-update active-thread list with a single Resume all action when native continuation is unavailable.

Related issues

This request is for a coordinated bulk maintenance operation across active threads.

Investigated with OpenAI GPT-6-Astra in the Codex harness; findings are from shipped source inspection, not an executed migration test.

Activity

  1. juliusmarminge commented on Oct 3, 2026

    @juliusmarminge
    Member

    Note

    Grok responding on behalf of Julius.

    Triage

    Thanks @kossoy for the careful source inspection and for being clear about what was and wasn't reproduced. I checked this on current main (31a9da179). Your reading of the staged nightly matches the source, and there's no second migration path that turns an in-flight V1 session into a V2 restart continuation. That gap comes from the documented protocol cutover rather than a regression in the restart setting. The bulk pause, update, and resume operation you describe isn't implemented today.

    What an update already does

    Settings → General → Continue threads after restarts (continueThreadsAfterServerUpdate, off by default, overridable per project) is the V2 mechanism. On a clean shutdown, prepareForShutdown in apps/server/src/orchestration-v2/ProviderRuntimeRecoveryService.ts records qualifying runs and enqueues provider-runtime.continue. After the process comes back, continueRestartedRun sends "Continue where you left off." for a cancelled source run that still has a strong native provider thread. Startup reconciliation can do the same for a run that's still running, or for a settled run whose background work was cancelled. Threads you already stopped stay stopped, because restartContinuationRun only accepts those cases.

    The old per-update flag isn't part of this. ServerEnvironment advertises threadRestartContinuation but leaves serverUpdateThreadContinuation unset, noting that the per-update request flag isn't wired into the V2 update RPC. The client only sends continueRunningThreads when that capability is true, and withRunningThreadContinuation isn't in the live update layer. So the update button doesn't capture an active-thread set, wait for a safe boundary, or report which threads resumed.

    A custom systemd unit also doesn't get the in-app updater unless it's the T3 background service (t3 service install). Restarting the process yourself still runs the shutdown finalizer, but only for V2 runs that qualify.

    The protocol 1 to 2 cutover

    Your notes are right, and nothing else fills the gap:

    • importedThread() in LegacyV1ThreadImporter.ts sets activeProviderThreadId: null and historyOrigin: "v1_import". Transcript rows get runId: null and no provider thread, and a message that was still streaming becomes an interrupted turn item. The importer doesn't import runs or provider sessions.
    • restartContinuationRun() then finds no run and no strong native provider thread, so continueRestartedRun() never runs for those threads.
    • The supported continuation is the first later message: shouldPrepareLegacyImportHandoff builds a handoff from the imported text and starts a fresh provider session. That's manual and one thread at a time.

    This is described in Threads from older T3 Code versions, Legacy orchestration migration, and Updating. Like you, this is from source and docs; the upgrade itself wasn't run.

    What would be new

    A single operation that captures the pre-update active set, pauses or interrupts it, resumes only that set, shows which threads came back, and lets you retry the rest (including a Resume all when native sessions can't be kept) would be a new workflow. The current setting resumes qualifying V2 threads quietly, with no status or retry surface, and doesn't drain work to a safe boundary first.

    #13740 and #12825 are different, as you said. The closest existing proposal is Discussion #14813 (install a desktop update when agents are idle), which doesn't cover remote/systemd updates, a captured active set, resume status and retry, or the V1 handoff case.

    If you'd like to push for the coordinated operation, Ideas is the best place. A maintainer will decide what to do with this issue. For this nightly jump, the supported path is to let running work finish (or accept the interruption), update, and send a new message on each thread you want to continue.

  2. locked and limited conversation to collaborators on Oct 11, 2026
  3. converted this issue into a discussion #18024 on Oct 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementRequested improvement or new capability.via-triageFiled through npx t3 triage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions