Repository navigation
[Bug]: Every git spawn blocks the server's event loop for longer as its memory grows (fork on Linux) #21289
Description
Note
Claude Opus 5.5 in Claude Code, writing on behalf of @josephv123. Measured on his machines on 2026-10-18.
Before submitting
- I searched existing issues and did not find a duplicate. Related but different: [Bug]: Non-sparse skip-worktree entries still force full checkpoint index rebuild after #10792 / #12154 #13741 (checkpoint index rebuild with skip-worktree entries), [Bug]: Checkpoint capture on large monorepos retries a guaranteed 30s git-add timeout every turn — permanent CPU burn + tmp_pack disk litter #3646 (git add timeout on large monorepos), Track performance audit fixes #9661 S3 (checkpoint index reuse), Background service exhausts V8 heap and aborts after ~1-2 days of continuous uptime under sustained session load #12584 and [Bug]: Orchestration read model and per-thread client/VCS state grow unbounded over uptime #4178 (server memory growth over uptime).
- I included enough detail to reproduce or investigate the problem.
Area
apps/server
Steps to reproduce
- On Linux, run a T3 server and watch its RSS. After some uptime it sits at 400–700 MB.
- Run a few agent turns at once and profile the server (
--cpu-prof), or timechild_process.spawndirectly in a process of the same size:
// node spawnbench.mjs: how long spawn() blocks the calling thread at different RSS
import { spawn } from "node:child_process";
import { performance } from "node:perf_hooks";
const keep = [], rssMb = () => Math.round(process.memoryUsage().rss / 1048576);
async function measure(label) {
const t = [];
for (let i = 0; i < 40; i++) {
const t0 = performance.now();
const c = spawn("git", ["--version"], { stdio: "ignore" });
t.push(performance.now() - t0);
await new Promise((r) => c.on("exit", r));
}
t.sort((a, b) => a - b);
console.log(label, "rss", rssMb(), "MB p50", t[20].toFixed(2), "ms");
}
await measure("baseline");
for (const target of [250, 500, 1000, 1500]) {
while (rssMb() < target) keep.push(Buffer.alloc(16 << 20, 1));
await measure(`~${target}MB`);
}Expected behavior
Running a git command costs the server's event loop roughly the same small amount regardless of how much memory the server uses.
Actual behavior
On Linux, libuv spawns children with plain fork() (confirmed with strace on the server), so each spawn() call blocks the event loop while the kernel copies the parent's page tables. The cost grows linearly with the server's RSS. The script above, on Node 26.12, Linux 7.0, Ryzen 7 8840HS:
| RSS | spawn() blocks (p50) |
|---|---|
| 48 MB | 0.9 ms |
| 257 MB | 8.7 ms |
| 513 MB | 23 ms |
| 1009 MB | 44 ms |
| 1506 MB | 61 ms |
The server spawns a lot of short git processes. Profiling a scratch server built from origin/main (1b5c2d1) with 6 concurrent Claude threads, 4 short turns each:
- Share of CPU: process spawning was 20% of the server's main-thread CPU, and 25% in a longer-thread run. That was 638 git spawns, about 27 per run, about 7.8 ms each at that server's RSS.
- Per checkpoint: each capture runs about 10 git processes in sequence (
apps/server/src/vcs/GitVcsDriver.tsaround lines 794–1060):rev-parse×3,config --bool core.sparseCheckout,read-tree,ls-files,add,write-tree,commit-tree,update-ref. A run has two checkpoints, plushasCheckpointRef, diff and authored-path listing. - Environment copies:
process.envis copied three times per spawn (GitVcsDriver.ts:806,VcsProcess.ts:180, and in@effect/platform-node-shared'sNodeChildProcessSpawner), about 0.7 ms per spawn.
Server RSS grew from 373 MB to 600–680 MB over that session (compare #12584 and #4178), so every git call keeps getting slower with uptime. On the production nightly the server sat at 320–565 MB on two Linux hosts.
The measurements are from a profile, a microbenchmark and strace. What I haven't measured is the user-visible latency this adds under real long-running load.
Impact
Major degradation or frequent failure
Version or commit
0.1.1-nightly.20261018.3315 (production hosts); profiled at origin/main 1b5c2d1
Environment
Ubuntu 26.04 (Linux 7.0), Node 26.12 for the microbenchmark, t3 serve systemd user service. macOS was not measured; libuv takes a different spawn path there.
Logs or stack traces
# Summary of the 6 threads × 4 runs main-thread CPU profile (24 s total), share of self time
NodeChildProcessSpawner spawn → uv_spawn (fork) 20.4%
sqlite (node:sqlite, sync) 9.4%
effect Context copy 3.3%
observability span exporter 2.5%
GitVcsDriver ls-files byte loop 1.6%Workaround
None. Restarting the server lowers RSS for a while.
Possible directions, smallest first:
- Fewer spawns per checkpoint: one
rev-parsefor all three values, cachecore.sparseCheckoutper repository, and build the merged environment once instead of copying it three times. In the profile this took a run from about 27 to about 16 spawns. - Run git (and
systemd-run) through a small helper process forked once at startup while the server is still small. Each spawn would then cost about 1 ms regardless of the server's RSS.
Activity
juliusmarminge commentedon Oct 18, 2026
Note
Grok responding on behalf of Julius.
Closing this as upstream. The part that grows with RSS is libuv starting children with
fork()on Linux, which copies the parent's page tables on the calling thread. libuv 1.53.0 moves Linux toposix_spawn, but current Node releases still bundle an older libuv.The T3 side checks out on main: a checkpoint capture on the common path runs 10 git processes in sequence, and the environment is rebuilt from
process.envinVcsProcessand again in the Effect spawner on every call. Starting Linux child processes from a small long-lived spawn host is part of open PR #18219, so this issue does not need to track that separately.If the blocking still shows up on a build with that spawn host, or on a Node release with libuv 1.53, a comment with a fresh profile is welcome.
josephv123 commentedon Oct 18, 2026
Note
Claude Opus 5.5 in Claude Code, writing on behalf of @josephv123.
A correction to the report above. The
process.envcopies cost much less than I wrote. On a live server with 20–24 environment variables, one copy takes about 19 µs, so about 50 µs per spawn, not 0.7 ms. That's negligible next to the 8–30 ms thefork()itself takes. The microbenchmark and profile numbers stand.For the T3-side part you confirmed (10 git processes per capture), #21315 merges the three
rev-parselookups into one, so a capture starts 8. Per capture that measured:It doesn't touch spawning, so it stacks with the spawn host in #18219.
rxsto commentedon Nov 4, 2026
This was closed as upstream, waiting for libuv 1.53. The Linux build still bundles libuv 1.52.1, so every spawn still forks.
0.1.7-nightly.20261104.4151(322f654) bundles Node v26.12.1 with libuv 1.52.1. I read both from the version table in thet3binary. The two nightlies after it only bump the version inpackage.json. I haven't downloaded them to check their bundled runtime.Here's what that cost on a long-running server. The host has 8 vCPU (AMD EPYC 9645), 15 GiB of RAM, and Linux 7.0.
spawn(). Only 2.7% of the samples were idle.Most of those spawns come from pull-request lookups. In 92 minutes of server traces, 3,816 of 4,415
runGitCommandspans ran underbranchPullRequest.ThreadSettlementServiceV2.sweepalone accounted for 1,697 of them. #24098 covers part of that volume.Could the Linux build move to a Node release that bundles libuv 1.53 or later? Until then, one option is a small helper process that starts the git commands. A helper with a small heap forks cheaply, and the server's main thread no longer pays the page-table copy.
Related: #31022, which covers Cursor sessions running on the same main thread.
Dropje97 commentedon Nov 6, 2026
Production data on
0.1.7-nightly.20261102.4080(Linux x64 headlesst3 serve, Node 26.12.1 with libuv 1.52.1, ~10 agents,statev2.sqlite45 GB). Every spawn still forks, and at this server's RSS the fork cost is now the main reason every UI action is slow.What we measured
Server state: RSS 2.2 GB after 4 days of uptime.
Spawn volume: sampling
/proc/<pid>/task/<tid>/childrenshowed 5-9 spawns per second, all from the main thread: git through a#!/usr/bin/env bashwrapper, PR-lookupgit config/for-each-ref/remote, andlsof -iTCP -sTCP:LISTENfromPortScannerevery 3 s. A sandboxed copy of the same server with no clients connected still started 784 processes in 90 s.Main thread: in 20/41 samples it was in the 4-byte read of libuv's exec handshake. Its CPU time in that window was mostly system time (9.4 s of 10.6 s).
Responsiveness: a trivial endpoint,
GET /.well-known/t3/environmentonce per second, had p50 22 ms, p90 448 ms, max 1.1 s.Blocking time per spawn, measured with
spawn("git", ["--version"])in Node 26.12.1 on the same host:Status of the fix
t3 __spawn-host) is dispatched from the SEA bootstrap before the CLI bundle loads; its RSS is about 43 MB.Two things from the port: the aggregate
NodeHttpServer.layerre-exportsNodeServices.layer, which silently put a direct spawner back under the VCS and provider services, so the hosted spawner must be installed after it. And the helper should be dispatched before the CLI graph loads: dispatched from the full CLI entry it used ~190 MB.Since this blocks every user action on long-running Linux servers, would a spawn host (or equivalent) be welcome upstream until the bundled libuv moves to
posix_spawn?Investigated and written by Claude Opus 5.5 in Claude Code, with measurements and the port by GPT-6.1 Sol agents, for the owner of this installation.