Repository navigation
[Bug]: Orphaned t3 serve processes remain after closing app sessions, causing port leaks and high memory usage #2614
Description
Before submitting
- I searched existing issues and did not find a duplicate.
- I included enough detail to reproduce or investigate the problem.
Area
apps/server
Steps to reproduce
-
Connect to a machine remotely through T3 Code.
-
Start a T3/Codex app session that launches a local
t3 serveprocess. -
Close the remote app/session, disconnect, or reconnect.
-
Start another T3/Codex app session.
-
Repeat this a few times.
-
Check for remaining local servers:
ps -eo pid,ppid,user,lstart,etime,rss,args | grep 't3 serve'
-
Check listening ports:
ss -ltnp | grep -E ':(377[0-9]|378[0-9]|379[0-9]|380[0-9])\b'
Expected behavior
When a remote T3 Code app/session is closed, disconnected, or reconnected, the associated t3 serve process and its child processes should be terminated.
Starting a new session should not leave old t3 serve processes running, and should not keep incrementing to new localhost ports because previous servers were leaked.
Actual behavior
I have only noticed this when using remote connections through T3 Code.
Old t3 serve processes appear to remain running after a remote app/session is closed, disconnected, or reconnected. My suspicion is that reconnecting starts a new local server but does not reliably shut down the previous one.
The old processes become orphaned with PPID=1, continue listening on localhost ports such as 3773, 3774, 3775, etc., and continue consuming memory.
After repeated remote sessions/reconnects, many stale t3 serve instances can accumulate and use several GB of RAM.
Impact
Blocks work completely
Version or commit
No response
Environment
No response
Logs or stack traces
Example stale process:
node /home/fionn/.npm/_npx/057fe8ede92e4d7d/node_modules/.bin/t3 serve --host 127.0.0.1 --port 3773 --base-dir /home/fionn/.t3
Process inspection showed old servers orphaned under PID 1:
ps -o pid,ppid,user,lstart,etime,rss,args -p <pid>
Example output:
PID PPID USER STARTED ELAPSED RSS COMMAND
5930 1 fionn Sat May 9 12:23:14 2026 05:41 175888 node .../t3 serve --host 127.0.0.1 --port 3773 --base-dir /home/fionn/.t3
Port inspection showed multiple local servers still listening:
ss -ltnp | grep -E ':(377[0-9]|378[0-9]|379[0-9]|380[0-9])\b'
Example output:
LISTEN 0 511 127.0.0.1:3773 0.0.0.0:* users:(("node",pid=5930,fd=21))
LISTEN 0 511 127.0.0.1:3774 0.0.0.0:* users:(("node",pid=6921,fd=21))
Aggregated memory usage from leaked instances:
ps -eo rss=,args= | awk '/t3 serve --host 127.0.0.1/ {sum+=$1; count++} END {printf "t3 serve: %.1f MiB across %d processes\n", sum/1024, count}'
At one point this showed dozens of `t3 serve` processes using several GiB of RAM.
System impact seen in kernel logs:
Out of memory: Killed process <pid> (qemu-system-x86)
The OOM event was not caused by one `t3 serve` process alone, but the accumulation of many leaked local servers contributed significantly to memory pressure.Screenshots, recordings, or supporting files
No response
Workaround
Manually list stale T3 servers:
ps -eo pid,ppid,user,lstart,etime,rss,args | grep 't3 serve --host 127.0.0.1'
Check which ports are still listening:
ss -ltnp | grep -E ':(377[0-9]|378[0-9]|379[0-9]|380[0-9])\b'
Kill stale/orphaned T3 servers for the current user:
pkill -f 'node .*t3 serve --host 127.0.0.1 .*--base-dir /home/fionn/.t3'
Safer manual cleanup, only targeting orphaned processes where PPID=1:
ps -eo pid=,ppid=,args= |
awk '$2 == 1 && /node .*t3 serve --host 127\.0\.0\.1 .*--base-dir \/home\/fionn\/\.t3/ {print $1}' |
xargs -r kill
After cleanup, verify:
pgrep -af 't3 serve --host 127.0.0.1'
ss -ltnp | grep -E ':(377[0-9]|378[0-9]|379[0-9]|380[0-9])\b'
A possible product-side workaround would be to have T3 Code detect existing t3 serve instances for the same --base-dir on reconnect and either reuse them or explicitly terminate stale orphaned ones before starting a new server.
Activity
juliusmarminge commentedon May 9, 2026
I am not sure i understand the issue. Can you clarify what commands and actions you run on either machine?
juliusmarminge commentedon Jul 20, 2026
Please provide the exact local or remote launch path, parent and child PIDs, command line, expected shutdown trigger, and the sequence used to close the app/session. Remote lifecycle ownership has changed since this report and those details are needed to identify a current finalizer bug.\n\n
fionn77 commentedon Aug 4, 2026
Thanks for following up, and sorry the original report was too broad. I reconstructed the May 8-9 setup from retained process snapshots and launcher state.
A correction first: this was the Windows desktop app managing an SSH remote. It was not an Android/mobile launch, and the original report was observational (accumulated leaked processes), not a controlled one-close/one-tree capture.
Controller and launch path
Gartlan-PC0.0.22C:\Users\fioei\AppData\Local\Programs\t3code\T3 Code (Alpha).exefionn@192.168.178.154:22(later saved as the aliasgartlanserver)/home/fionn//home/fionn/.t3t3 servein a shell. Selecting/connecting to that SSH environment in the desktop app launched it.The retained generated runner for the original IP-keyed target was:
After the same target was saved under the
gartlanserverSSH alias, its state key/path was:The remote runner was using
t3@0.0.22and launched the server withT3CODE_NO_BROWSER=1, loopback host, a scanned port starting at 3773, and--base-dir /home/fionn/.t3.One potentially relevant local detail: at first the app could be invoked both manually and by this plain Windows Startup shortcut:
I noticed that after login I could manually start T3 Code and then have the Startup shortcut invoke it again. I changed that shortcut to an "already running" guard on May 8. Leaked remote processes had already been observed before that change, so duplicate/local startup may have been a contributing path but does not account for the whole history by itself.
Captured process tree and commands
A complete remote chain captured from the same affected version/configuration was:
Other older leaked children from that run had already lost the npm/shell wrappers and were reparented directly to PID 1, for example:
The May 9 snapshot included:
I did not capture the pre-orphan parent PIDs for 5930/6921. By the time I inspected them their parent was already PID 1, so I cannot provide the original launcher PIDs for those two without guessing.
Expected shutdown trigger and the actual sequence
The shutdown trigger I expected was either:
That should close the local
ssh -N -L ...tunnel and run the remote managed-server stop/finalizer. I no longer claim that merely leaving or closing a Codex thread should necessarily stop an environment-level server; the original phrase "close the remote app/session" conflated thread navigation with environment lifecycle.The sequence used during the accumulated observation was:
fionn@192.168.178.154SSH environment.node ... t3 servechildren remained.psandss.The important limitation is that the May 9 PIDs were not tagged to one isolated UI-close action. I cannot honestly say whether PID 5930 specifically survived explicit Disconnect, app Quit, or an app restart/duplicate launch. The confirmed facts are the launcher path/command, the process trees above, and that the older node children were still listening after their wrappers/owners were gone.
Since lifecycle ownership has changed, I agree the most useful next step would be a fresh controlled capture on the current build: tree before launch, tree after explicit Disconnect, and tree after a full app Quit. I can do that if this historical detail is not enough to map the old report to the current finalizer.
acomarcho commentedon Sep 3, 2026
Independent reproduction on 0.0.35, plus what looks like the concrete root cause for the orphaning described here. The original report's suspicion ("reconnecting starts a new local server but does not reliably shut down the previous one") is correct, and the reason it cannot shut it down is a PID recorded one layer too high.
Root cause: the recorded PID is the npm wrapper, not the server
REMOTE_LAUNCH_SCRIPTinpackages/ssh/src/tunnel.tsstarts the remote server and records$!:$RUNNER_FILE(run-t3.sh) onlyexecs straight into the server when a globalt3is on PATH. When it is not, it falls through to the npx path:npxisnpm exec, which does not exec-chain into the resolved CLI. It spawnssh -c "t3" ..., which spawns the actual node process. So$!is thenpm execprocess, two layers above the server that actually binds the port and opens SQLite.Captured from a live remote host while it was in this failure state:
Every
kill "$REMOTE_PID"in the launch script therefore signals 2618056 and never reaches 2618095.Reproduction of the signal behaviour
Minimal package in the same three-layer shape (
nohup env ... runner.sh &->exec npm exec->sh -c-> node listener), then SIGTERM the pid that$!returned:Result:
npm does not forward the signal through the intervening
sh -c, so the server survives, reparents to init, and keeps both its listening socket and its SQLite connection.Why this yields the incrementing ports
Once the wrapper is dead and the server is not, the launcher's own bookkeeping is wrong in a way that guarantees escalation:
kill -0 "$REMOTE_PID"now fails, so the script treats the state as stale and clears$PID_FILE/$PORT_FILE/$MANAGED_FILE.pick_portscans upward from the last recorded port, window 200. The orphan still holds the previous port, so it returns the next one.--base-dir "$DEFAULT_SERVER_HOME", so it opens the samestate.sqliteas every orphan before it.On the host I looked at this produced nine distinct ports over the logs, 3773 through 3781, five of them live at once, all against one database. Contention then surfaces as repeated
PersistenceSqlError ... OrchestrationEventStore.append:insertwithError: database is locked(463 occurrences across two log files), and codex threads failing withthread <id> already has an active writerbecause the provider is still attached to a server the desktop has abandoned.Worth noting the contention is entirely from duplication, not data volume: with a single server alive on the same 3.2 GB database,
GET /and/.well-known/t3/environmentboth answer in 1-5 ms.Fix material already present but unused
server-runtime.jsonrecords the real server pid, andresolve_default_runtime_portin the same script already reads it and liveness-checks it withprocess.kill(pid, 0). The kill path just prefers the wrapper:Preferring the runtime-file pid, or having the server write its own pid to
$PID_FILEafter binding, would make the existing teardown work. A process-group kill would be an alternative, sincenohup ... &does not create a new process group on its own.Still current
packages/ssh/src/tunnel.tsis byte-identical betweenv0.0.35andmainat5b8445b7a777, so this is unchanged as of 0.0.38.Related: #5749 covers why the replacement server gets started in the first place (the 2s reuse readiness budget), and #6097 covers the missing single-ownership check on the database. This issue is the third leg: why the old one never goes away.
Environment
t3@0.0.35), no globalt3on PATH so the npx path is takent3 service statusreports not installed), so every server is launcher-managedFiled via
t3 triage. Investigation and reproduction by Claude Opus 5 (1M context) running as the triage agent in Claude Code.