[Bug]: On Windows, every git start blocks the server's event loop, freezing it for seconds

Before submitting

I searched existing issues and did not find a duplicate.
I included enough detail to reproduce or investigate the problem.

Area

apps/server

Steps to reproduce

On Windows with Git for Windows in C:\Program Files\Git, time the synchronous part of starting
git.exe from Node, one start a second, the way the server's sweeps spread them out:

// node time-git-spawn.cjs
const { spawn } = require("node:child_process");
const git = String.raw`C:\Program Files\Git\mingw64\bin\git.exe`;
let runs = 0;
const start = () => {
  const before = performance.now();
  spawn(git, ["--version"], { stdio: "ignore", windowsHide: true });
  console.log(`${(performance.now() - before).toFixed(0)} ms`);
  if (++runs < 12) setTimeout(start, 1000);
};
start();

At times, back-to-back starts were fast while spaced ones stayed slow (see below), so a
benchmark that starts git back to back can miss this.

Run the T3 Code server on the same machine with a few projects whose threads sit on branches,
and leave it running. Pull request discovery and status checks start git processes every minute.
Watch event-loop-stalls.ndjson in the server's logs, or use the app while a sweep runs.

Expected behavior

Starting a git process doesn't hold the event loop long enough to notice. Websocket requests,
provider events and the UI stay responsive while background sweeps run.

Actual behavior

On the affected PC, how long a synchronous spawn() of the installed git.exe takes varies over
time:

One afternoon, in interleaved back-to-back runs of 30-60 starts per program, each waiting for
the previous one to exit, it typically took 486-582 ms per start (worst 2.1-2.3 s). A
byte-identical copy of git.exe and its four DLLs in a fresh folder took 10-14 ms (worst
217-390 ms).
That evening, the same back-to-back runs took 4-5 ms for the original and the copy alike, while
starts spaced a second apart still took 0.5-2 s. That is why the snippet spaces its starts.

Hard links and the program's location were checked and ruled out. So were Smart App Control (off),
attack surface reduction rules (none) and Image File Execution Options entries (none for
git.exe or node.exe). Antivirus scanning of the heavily used original would fit the data, but
that is not proven. A short trial excluding the file from Defender didn't settle it: the
back-to-back starts had already got fast before the exclusion went on, and the server kept
freezing while the file was excluded, with stalls of 5.3 s and 2.1 s.

The server starts git often: in a 27-minute trace, pull request sweeps alone started 11,182
processes, 11,180 of them git, about 400 a minute. Every start runs on the event loop, so
overlapping starts queue there. 60 git reads at the git semaphore's concurrency of 8 stalled the
event loop for up to 4.4 s and 4.5 s in two rounds. While that happens the app freezes, and
websocket requests and provider events wait behind the stall.

Impact

Major degradation or frequent failure

Version or commit

Freezes seen on T3 Code (Nightly) 0.1.3-nightly.20261024.3643. Git still starts through Node's
spawner on the event loop on main @ 7bafc40.

Environment

Windows 11 Pro (build 26300), Git for Windows 2.55.0 in C:\Program Files\Git, Microsoft
Defender on. The timings above used Node 24.22.1.

Workaround

None that holds up. A fresh copy of git.exe was faster only in the afternoon runs, and pointing
T3 Code at a copy is fragile. Settling or archiving threads whose pull requests are merged
lowers how many git processes the sweeps start (#24098), not what each start costs.

Independent Windows evidence: native spawn stalls and a local-primary heartbeat timeout

I can independently confirm the native-spawn blocking symptom, with a sampled backend stack and a separately captured local connection failure. Posting here rather than opening a duplicate.

Environment / build disclosure: Windows 11 Pro, build 26300; desktop executable reports T3 0.1.2; shipped Electron 44.9.3 / Node 24.22.0; Git 2.40.1.windows.1. The server bundle is locally modified with asynchronous Windows executable lookup and SQLite read routing. This is not a clean-stock reproduction. Its packaged Effect spawner still calls native NodeChildProcess.spawn(...) directly on the backend event loop. The user regularly runs dozens of agent threads, but the exact active count during this capture was not established.

Observed evidence

A 100 Hz V8 profile of the live backend on 2026-10-26, approximately 01:20:59–01:22:00 UTC:

Measurement	Result
Profile duration	61,081.8 ms
Native `spawn` sampled wall time, summed across its leaf nodes	46,298.9 ms (~75.8%)
Longest consecutive run of samples in native `spawn`	7,381.2 ms
`runGitCommand` spans overlapping the capture	41
`processRunner.runProcessCore` spans marked `process.command=git`, overlapping the capture	7

The sampled call path was:

native spawn
  node:internal/child_process -> spawn
  node:child_process -> spawn
  @effect/platform-node-shared/NodeChildProcessSpawner.js
    -> NodeChildProcess.spawn(command.command, command.args, spawnOptions)
  Effect runtime

The native call site was verified in the running installation's server bundle. The public process runner still delegates launch to the Effect spawner.

These are statistical wall-time samples, not CPU utilization: they can include native waiting/descheduling. Consecutive samples in the same frame may cover back-to-back calls, so 7.38 seconds is not claimed to be one measured spawn call. The Git spans overlap and are not an exact or additive process-launch count. Profiler startup acknowledgement time is not included in the spawn attribution.

To check whether slow launch exists without profiling, a separate finite fixture using the same shipped Electron executable launched only git --version:

Phase	Synchronous spawn return, launch 1 / 2	Total command elapsed, launch 1 / 2
No profiler	621.9 / 1,028.0 ms	3,180.9 / 3,858.6 ms
100 Hz profiler	656.8 / 411.2 ms	2,338.8 / 3,536.9 ms

This is only two launches per phase under uncontrolled host load, not a benchmark proving zero profiler effect. It does show that slow synchronous launch also exists with the profiler off.

Real disconnect: failure started before focusing T3

In a later window, with no CPU capture active, an out-of-process probe and foreground-state sampling captured:

UTC	Observation
01:39:46 / 01:39:49	Local metadata endpoint responded in ~16 ms
01:39:59.434	TCP connected immediately, but HTTP response headers timed out after ~828 ms
01:40:03 / 07 / 10	Repeated header timeouts, immediate TCP connection
01:40:12.887	User switched from another window to the already-open T3 window
01:40:18.766	`EnvironmentSupervisor.connectionLost`, local primary target, transport failure; detail classified as `stopped responding for ...`
After switching away again	Independent probe failures continued

The client ping timeout is 15 seconds, and the disconnect path distinguishes this ping-timeout detail.

For this event, backend response failure preceded foreground return by 13.45 seconds. The observed “Syncing messages…” then “reconnecting” sequence therefore does not establish that focusing the window caused the failure.

The metadata endpoint also performs asynchronous filesystem work, so its timeout alone is not a pure event-loop measurement. The CPU profile and disconnect are different windows; we have not proven that a particular Git launch caused this exact heartbeat loss.

Suggested T3-side investigation / changes

Isolate native child creation from the event loop serving RPC and heartbeats. Evaluate a long-lived launch worker/broker rather than merely wrapping spawn() in a Promise/Effect. Preserve stdout/stderr streaming, backpressure, exit status, cancellation, and process-tree ownership. A launch worker may block on Windows, but the primary backend should remain responsive.
Reduce duplicate background Git launches per worktree / branch. Coalesce in-flight read requests, reuse valid cached results, and stagger sweeps. Keep freshness checks before settlement or other state-changing decisions. This complements [Bug]: Threads whose branch PR is merged or closed re-read git state every minute #24098 and should not require lowering user agent-task concurrency.
Measure creation delay separately from command runtime and queue wait. Record bounded aggregates for native spawn return latency, pending launches and event-loop lag. Increasing RPC concurrency or extending the 15-second timeout alone does not remove synchronous blocking.
Test responsiveness under deliberately delayed child creation. Assert that independent RPC / heartbeats remain responsive while the launch worker is delayed, and that command output, cancellation and cleanup retain their behavior.

A small spawn-host direction appeared in #18219, but that PR is closed/unmerged and described Linux support; I have not verified a Windows equivalent. #21289 concerns Linux fork/RSS cost, which should not be assumed to explain Windows native creation latency.

The underlying Windows cause remains unknown; these measurements do not establish Defender, disk, memory growth, or a T3 database lock as the cause. No raw profiles, conversation contents, credentials, environment-variable values, private repository paths or host identifiers are attached.

Prepared and submitted by Codex at the user's request. No running T3 tasks were stopped or restarted for this investigation.

fix(pingdotgg#25245): share concurrent Git metadata reads

chore(pingdotgg#25245): clear desktop release check failures

fix(pingdotgg#25245): reduce background repository probe storms

fix(pingdotgg#25245): call Gemini directly for approval explanations

fix(pingdotgg#25245): encode approval requests with the app schema codec

fix(pingdotgg#25245): move Windows Git launches off the backend thread

fix(pingdotgg#25245): route remaining buffered Windows Git calls thro…

fix(pingdotgg#25245): move background Windows launches off the server…

Another Windows datapoint: stalls about once a minute, and what `detached: true` does and doesn't fix

Posting here rather than in a new issue. It's the same mechanism on a different machine, plus one partial mitigation measured inside T3.

Environment / build disclosure: Windows 11 Enterprise (build 26200), 16 logical cores, 62 GB RAM, Git for Windows 2.49.0, T3 Code (Nightly) 0.1.7 and 0.1.8 builds. It's a managed corporate laptop. Endpoint security software inspects every process creation, so each launch costs more than on a clean machine. The last section uses a locally patched server bundle (a preload that changes spawn options for some git calls). Everything before that section is stock.

What the stall log shows (stock build)

From 08:30 to 13:00, event-loop-stalls.ndjson logged 25–46 stalls per 30 minutes. The median stall was 2.5–3.9 s and the worst was 10.9 s. Every record has utilization: 1, with 2–6 s of system CPU per window.
In one 27.7-minute trace window, the server ran 484 processes (17.5/min), 453 of them git. The processRunner.runProcessCore spans for git had a median of 4.0 s and a p90 of 7.3 s. In the same minutes, a plain cmd /c exit from a terminal took 0.26–0.77 s. So most of each span is probably time spent queued behind the blocked loop, not the git work itself.
That window also re-ran the full source-control discovery set four times (gh, glab, az, jj and three other CLI probes). That adds launches at exactly the moment launches are expensive.
The per-launch cost on this machine also varies a lot within one boot. It's under a second most of the time and 3–10 s in bad stretches. That matches the "varies over time" observation in the original report.

The SQLite spans are mostly queueing, not database time

This matters for anyone who lands here from #26224 / #30065 / #29468.

In the same trace, sqlite spans had a p90 of 2.4 s and a max of 12.7 s, and BEGIN/COMMIT averaged about 300 ms each.
I ran the same read statements from a separate read-only connection to the same statev2.sqlite, while the server was stalling. They took 0.3–23 ms.
So on this machine the slow DB spans mostly measure the blocked event loop, not SQLite.

The database had also grown to 5.56 GB, with 46% free pages and auto_vacuum = NONE (a migrated install, as in #30066). I ran VACUUM offline with auto_vacuum = INCREMENTAL, which shrank it to 2.90 GB. The stalls continued afterwards: 11 in the first ~12 minutes after restart, median 3.7 s, worst 6.9 s. The restart also re-applied the git patch below, so neither change on its own stopped the stalls.

Partial mitigation: `detached: true` for read-only git calls (locally patched)

On win32, the Effect Node spawner defaults to detached: false, windowsHide: options.windowsHide ?? !detached (@effect/platform-node-shared 4.0.5, nodeChildProcessSpawner). libuv turns windowsHide with piped stdio into CREATE_NO_WINDOW, and that still creates a console host for every git child.
With detached: true, the child gets no console, so no console host is created.

I measured this outside T3 first. A console-less Node parent (as T3's backend is) timed spawn → exit, interleaved and rotated. The machine was healthy, early in the boot:

Command	`windowsHide` (today)	`detached`	Change
`cmd /c exit` (n=20)	1,165 ms median	413 ms	−65%
`mingw64\bin\git.exe --version`	1,283 ms	698 ms	−45%
`Git\cmd\git.exe` shim `--version`	1,890 ms	1,862 ms	none: the shim starts the real git, which gets a console anyway

Then I tried it inside T3, with a preload that sets detached: true only for read-only leaf subcommands (rev-parse, show-ref, for-each-ref, status, ls-files, config --get, worktree list, …) run through mingw64\bin\git.exe.

T3's own runProcessCore git spans dropped from a median of 1.24–1.42 s to 0.83–0.84 s (−33 to −41%). The p90 improved less (−5 to −22%).
These were back-to-back sessions, not a controlled A/B.
The stalls don't go away. The spawn() call is still synchronous on the event loop; it's just shorter.

One hazard: detached removes the console only for the direct child. A console-less child that starts console programs (hooks, credential helpers, ssh, ! aliases) gives each of them a new console. cmd /c ping under detached took 17–19 s in both runs. So detached only suits leaf read-only calls, not git calls that can start helpers.

What would help on Windows

A Windows version of fix(server): isolate Linux process launches from the RPC event loop #30209's approach: launch through a helper process so spawn() can't block the RPC loop. Of the fixes in this thread and its linked PRs, it's the only one that removes the stalls rather than shortening them.
In the meantime, detached: true (and mingw64\bin\git.exe, which preferGitForWindowsBinary already selects) for read-only leaf git calls. That cut the median launch cost by about a third to a half here.
Don't re-run the full source-control discovery probes on every reconnect while launches are slow. Otherwise slow launches cause a reconnect, the reconnect causes more launches, and those make launches slower.

I can share the trace-analysis scripts (Node, read the server.trace.ndjson* and stall files) if they would help.

Note

🤖 Claude Opus 5.5 responding on behalf of Elliot

Windows datapoint: CPU contention turns the per-start block into seconds, and a priority A/B isolates it

Setup:

Windows 11, 16 logical CPUs, 0.1.8-nightly.20261109.4399.
148 worktrees and 403 unarchived threads.
Antivirus real-time scanning off.

Probe: a separate, nearly idle Node process on the same host (monitorEventLoopDelay plus a 100 ms ticker). Every 2 s it times only the synchronous part of spawn('git', ['--version']).

A/B test: the same workload both times, two Playwright suites at 8 workers each for 240 s, with T3 left running. Only the test processes' priority class changed:

Test processes at	Host CPU	Run queue median	T3 `/.well-known` median / max	Probe max loop gap	Probe sync spawn p95 / max	Tests completed
Normal	100%	74	760 / 1652 ms	1131 ms	407 / 854 ms	1365
BelowNormal	79%	28	20 / 28 ms	31 ms	15 / 22 ms	1274

With the host idle, the same sync spawn takes 4–10 ms.
With antivirus real-time scanning on, an earlier session on this machine measured 134–550 ms per start.

Meanwhile the server starts about 300 git processes a minute while otherwise idle (runGitCommand about 309/min, readConfigValue about 135/min, from PR lookup and settlement; see #24098). So under contention the main thread spends seconds per second inside CreateProcess.

Takeaway: the stall is the main thread competing for CPU and process-creation time with everything else on the host. Moving spawns off the event loop (#25245) and cutting the spawn volume (#24098) would both decouple T3's responsiveness from host load. Lowering the priority of other workloads is a user-side workaround, not a fix.

Note

🤖 Claude Opus 5.5 (Claude Code) responding on behalf of Raphael

Windows datapoint: the stall at startup makes the desktop give up on its own backend

Same mechanism on another machine. One new consequence: the desktop's 180 s readiness check fails while the server's event loop is busy.

Environment: Windows 11 Pro (build 26200), 24 logical CPUs, 64 GB RAM, Git for Windows 2.55.0.windows.3, T3 Code (Nightly) desktop 0.1.9-nightly.20261111.4501. Defender real-time scanning is on. About 28 worktree directories across a few projects.

What happened: On the first launch after the nightly update, the backend started (port 3773) but never answered /.well-known/t3/environment in time. server-child.log:

backend child process failure output end
Timed out after 180000ms waiting for desktop backend readiness at http://127.0.0.1:3773/.well-known/t3/environment.

During those 3 minutes, every request to the environment endpoint failed while waiting on a small file read:

WARN environment api request failed
  errorTag: 'Interrupt'
  InterruptError: All fibers interrupted without error
    at ServerSecretStore.get
    at config.readAgentActivityPublishingActive
    at environment.metadata.descriptor

event-loop-stalls.ndjson: There are dozens of entries over roughly 55 minutes. Each 30 s sample shows utilization 0.9 to 1.0, delayMaxMs of 2 to 8 s and cpuSystemMs of 15,000 to 31,000. That means the server process spent most of each window in kernel time:

{"time":"…T05:16:33Z","delayMaxMs":7674,"utilization":1,"cpuUserMs":468,"cpuSystemMs":30844,"rssMb":438}
{"time":"…T05:18:12Z","delayMaxMs":8043,"utilization":1,"cpuUserMs":265,"cpuSystemMs":26281,"rssMb":429}

server.trace.ndjson, first 15 minutes after launch:

Span	Count	Total	Max
`runGitCommand`	784	3,492 s	23.1 s
`GitManager.rememberPrLookupGitState`	212	11,511 s	130 s
`GitVcsDriver.readConfigValue`	213	9,246 s	93 s
`sqlite`	569	1,354 s	39 s
`shell.resolveCommandPath`	168	1,884 s	445 s
`ServerSecretStore.get`	109	554 s	19 s

StorageCleanup.cleanWorktrees (inside StorageCleanup.sweep) and ThreadPullRequestServiceV2.synchronize both started at boot and ran for 979 s and 839 s. Later in the session, git starts peaked at 389 a minute. Across the last ~2 hours of traces there were 3,922 runGitCommand spans totalling 6,265 s.

Windows recorded no application crash events, and the renderer never logged render-process-gone. To the user it looked like a crash: the app reported a backend failure and was relaunched several times that day.

Suggestion beyond the spawn cost itself: At startup, either hold back the PR and worktree sweeps until the backend reports ready, or answer /.well-known/t3/environment without waiting behind them. Otherwise a git-heavy startup reads as a dead backend.

Navigation Menu

[Bug]: On Windows, every git start blocks the server's event loop, freezing it for seconds #25245

Before submitting

Area

Steps to reproduce

Expected behavior

Actual behavior

Impact

Version or commit

Environment

Workaround

Independent Windows evidence: native spawn stalls and a local-primary heartbeat timeout

Observed evidence

Real disconnect: failure started before focusing T3

Suggested T3-side investigation / changes

Another Windows datapoint: stalls about once a minute, and what `detached: true` does and doesn't fix

What the stall log shows (stock build)

The SQLite spans are mostly queueing, not database time

Partial mitigation: `detached: true` for read-only git calls (locally patched)

What would help on Windows

Windows datapoint: CPU contention turns the per-start block into seconds, and a priority A/B isolates it

Windows datapoint: the stall at startup makes the desktop give up on its own backend

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Bug]: On Windows, every git start blocks the server's event loop, freezing it for seconds #25245

Description

Before submitting

Area

Steps to reproduce

Expected behavior

Actual behavior

Impact

Version or commit

Environment

Workaround

Activity

Enmani commented on Oct 26, 2026

Independent Windows evidence: native spawn stalls and a local-primary heartbeat timeout

Observed evidence

Real disconnect: failure started before focusing T3

Suggested T3-side investigation / changes

shaded0 commented on Nov 6, 2026

Another Windows datapoint: stalls about once a minute, and what detached: true does and doesn't fix

What the stall log shows (stock build)

The SQLite spans are mostly queueing, not database time

Partial mitigation: detached: true for read-only git calls (locally patched)

What would help on Windows

ElliotDrel commented on Nov 9, 2026

Windows datapoint: CPU contention turns the per-start block into seconds, and a priority A/B isolates it

raphvallee commented on Nov 11, 2026

Windows datapoint: the stall at startup makes the desktop give up on its own backend

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Another Windows datapoint: stalls about once a minute, and what `detached: true` does and doesn't fix

Partial mitigation: `detached: true` for read-only git calls (locally patched)