[Bug]: Remote (SSH-launched) server dies on client disconnect, delegated subagents reported as cancelled #15608

Activity

1. juliusmarminge commented on Oct 4, 2026

   

   juliusmarminge

   on Oct 4, 2026

   Member

   More actions

   Note

   Grok responding on behalf of Julius.

   Triage

   Thanks for the careful report, @mb-hermes-bot. The sshd timestamps next to the server log made the order of events easy to follow.

   What I found

   I checked current main (5a96895a85). There are two separate behaviors here, and the nightly you're on has both.

   1. Quitting the desktop app stops the SSH-managed server. Your timeline fits a desktop shutdown, not a network drop or a SIGHUP from the launch session. The server had already outlived the launch SSH because it was started with nohup. It was interrupted at 11:21:40, and sshd logged disconnected by user for the two sessions only afterward, at 11:21:42 and 11:21:45.

   When the desktop app quits, DesktopApp's scope (apps/desktop/src/app/DesktopApp.ts) closes SshEnvironmentManager, and the tunnel finalizer in packages/ssh/src/tunnel.ts calls stopRemoteServer(). The stop script SIGTERMs the recorded PID unless ~/.t3/ssh-launch/<state>/managed is external. An explicit Disconnect and turning the environment off go through the same stop, while a server the launcher found already running is left alone.

   This is the same lifecycle as #10934, which was closed as completed on 2026-10-02 after #9843 made the finalizer stop the real t3 serve instead of an npm wrapper. That finalizer call is still there. #10951 tried keeping the remote server running across quit and was closed unmerged. Right now, quit and window-close stop a managed server, while the user docs only mention removing the connection.

   Continue threads after restarts in Settings doesn't keep this work alive. It's off by default, and even when it's on, shutdown cancels the live provider session first, so any continuation is a new turn.

   Workaround: on the remote host, install the background service (t3 service install) and reconnect, so the desktop adopts that process as external and the stop script leaves it running. You'll also need user lingering enabled, or the user service stops when you log out (see the background-service docs).

   2. A shutdown is recorded as a plain cancel. On graceful SIGTERM, ProviderRuntimeRecoveryService.reconcile("shutdown") (apps/server/src/serverRuntimeStartup.ts) marks the in-flight run status: "cancelled" and only sets completedAt. The message Cancelled because the server shut down before the provider work completed. is stored on the effect outbox row, not on the run, and task_status has no reason field. For a run that didn't fail, subagentResultForRun uses the last assistant message as summary, which is why you saw the child's last progress update with workState: "result_available". An intentional task_cancel ends in the same status. An interrupted status exists, but shutdown doesn't use it. So after you reconnect, neither the parent nor task_status can tell a server shutdown apart from a cancel.

   Likely fix area

   • The tunnel finalizer in packages/ssh/src/tunnel.ts and the SshEnvironmentManager teardown, depending on how quit and window-close should treat a managed remote server (the question [Bug]: Restarting the desktop client breaks running remote Codex threads over SSH #10934 left open).
   • ProviderRuntimeRecoveryService.reconcile("shutdown") and subagentResultForRun. Options include recording shutdown as interrupted, or carrying a cancel reason through to the run and task_status.

   A maintainer will decide on the fix direction.
2. juliusmarminge

   added

   bugSomething is broken or behaving incorrectly.Something is broken or behaving incorrectly.

   via-triageFiled through npx t3 triageFiled through npx t3 triage

   on Oct 4, 2026
3. DigitalWestern commented on Oct 6, 2026

   

   DigitalWestern

   on Oct 6, 2026

   More actions

   Same symptom here, plus two findings not covered above: the stop also fires on wake from sleep (not only on quit), and the documented workaround (t3 service install) did not help on its own, even with the service already installed and lingering enabled.

   Environment

   • Client: T3 Code Nightly desktop 0.0.46-nightly.20261005.2689 (later 2702), macOS arm64
   • Remote: Fedora 44, linux-x64, SSH connection target. t3code.service installed, Linger=yes, running 0.0.46-nightly.20261004.2657
   • Provider: Claude (Opus 5.5)

   1. Wake from sleep stops the remote server too

   When the Mac wakes, the stale tunnel fails the 2 s readiness probe in ensureTunnelEntry. That path calls closeTunnelEntry, which runs the entry finalizer, which calls stopRemoteServer(). No quit is involved.

   From the desktop trace (span chain desktop.ipc.sshEnvironment.ensureEnvironment > ssh/tunnel.ensureEnvironment > ssh/tunnel.ensureTunnelEntry > ssh/tunnel.closeTunnelEntry > ssh/tunnel.stopRemoteServer), on 2026-10-05 in UTC:

   Remote server stopped (stopRemoteServer)	Relaunched (launchOrReuseRemoteServer)
   16:34:06	16:34:14
   17:46:14	17:46:24
   19:01:42	19:01:45
   20:11:49	20:11:57
   22:09:22	22:25:26
   01:48:53 (Oct 6)	01:48:55

   There were at least 11 stale-tunnel stops that day (sleep/wake or network blips). The user returned to an interrupted turn and had to send "continue". The agent then reported that "the T3 restart killed" its background watcher.

   Code at v0.0.46-nightly.20261005.2689 (3e6b45028cee):

   • packages/ssh/src/tunnel.ts:1609-1631: stale-entry branch calls closeTunnelEntry(entry)
   • packages/ssh/src/tunnel.ts:1541-1580: entry finalizer calls stopRemoteServer() whenever the entry is still the registered one

   Same logic on current main.

   2. Why the service workaround did not take effect

   The service was already running on 127.0.0.1:3773. The desktop still ran its own managed server on :3774, with the same --base-dir ~/.t3 (so the same database), and stopped that one on every wake and quit. What I found:

   1. The managed server owns the shared discovery file. The launch script detects an external server only through ~/.t3/userdata/server-runtime.json (tunnel.ts:627-662). Every t3 serve writes that file on startup. Once a managed server was running, the file pointed at :3774, so the service was never found.
   2. Any server deletes server-runtime.json on shutdown, even when the file describes a different server (apps/server/src/server.ts:722-728, unconditional clearPersistedServerRuntimeState). I pointed the file at the service, then stopped the managed server. Seconds later the desktop launched a fresh managed server and the file described it again. That matches the dying server deleting my edit.
   3. A reconnect during a service restart launches a managed server. I ran systemctl --user restart t3code.service. The desktop reconnected during the ~1 s the service was down, found nothing on :3773, and launched a new managed server on :3774.

   Together, a managed server, once started, keeps replacing itself, and the installed service stays unused.

   Recovery that worked

   1. Stop the managed server and wait for it to exit (it deletes server-runtime.json).
   2. Immediately rewrite server-runtime.json with the service's pid, port 3773, and origin http://127.0.0.1:3773.
   3. Set ~/.t3/ssh-launch/<key>/managed to external and port to 3773, and remove pid.

   After that the desktop adopted the service as external. A lid-close test kept the turn running.

   Possible fix directions

   • Only clear server-runtime.json on shutdown if its pid matches the exiting process.
   • Don't stop the remote server on a stale-tunnel reconnect. Rebuild only the tunnel. Reserve stopRemoteServer() for an explicit Disconnect.
   • Prefer a running t3code.service over launching a managed server, for example by checking the unit before pick_port.

   ---

   Triage by Claude Opus 5.5 (Claude Code) via t3 triage, reviewed by the reporter.
4. t3dotgg commented on Oct 21, 2026

   

   t3dotgg

   on Oct 21, 2026

   Member

   More actions

   Note

   🤖 Claude Opus 5.5 responding on behalf of Theo

   Thanks for the report. One part of this is fixed on main in #19432. Please test on the nightly built from main on 2026-10-21, or on the next stable release. One part is still open: desktop quit still stops the managed remote server; shutdown is still reported as cancelled. If that part still matters to you, please open a new issue for that part only.
5. t3dotgg

   closed this as completedon Oct 21, 2026