Skip to content

Windows: orphaned git.exe is hung pre-main in detect_msys_tty/NtQueryObject; the wedged process is a grandchild of the cmd\git.exe shim and escapes direct-child cleanup #41783

Open

Description

@Evan-Haug

What version of Codex CLI is running?

codex-cli 0.151.0-alpha.7.2, run via codex.exe app-server launched by the Codex Windows desktop app (package OpenAI.Codex_26.825.6671.0_x64).

What subscription do you have?

Omitting; not relevant to this report.

Which model were you using?

Not model-dependent. The leak is in the git probe path.

What platform is your computer?

Windows 11 Pro 10.0.22631 x64. Git for Windows 2.39.2.windows.1 at C:\Program Files\Git.

What issue are you seeing?

Codex leaves git.exe processes alive forever on Windows. Same symptom as #17229 and #29110, but I attached a debugger to the live orphans and got the actual root cause, which as far as I can tell has not been posted before. It is not a polling-frequency problem and it is not a child blocked writing to a stdout pipe.

The hung git is frozen before main() runs.

A non-invasive cdb -p <pid> -pv -c "~*k" attach to six orphans that had been alive 6.8 hours gave the identical stack on every one:

ntdll!NtQueryObject+0x14
git+0x285702
git+0x2864d7
git+0x284647
git+0x12ef      <- wmain
git+0x1406      <- CRT entry
KERNEL32!BaseThreadInitThunk+0x1d
ntdll!RtlUserThreadStart+0x28

Git for Windows ships no PDBs, so I identified the frame from the call site itself:

mov     r9d, 3FEh        ; ResultLength arg == sizeof(buffer) - 2, buffer is BYTE[1024]
lea     r8,  [rsp+40h]   ; ObjectInformation
mov     rcx, rbx         ; Handle
mov     edx, 1           ; ObjectInformationClass = ObjectNameInformation
call    git+0x2b3f80     ; NtQueryObject
test    eax, eax
js      <bail>
cmp     dword ptr [rsp+3Ch], 0Fh

That is verbatim detect_msys_tty() in git-for-windows compat/winansi.c, called from winansi_init() at startup. It runs against fd 0, 1 and 2 whenever the handle is a pipe, to sniff for an MSYS2 pty name. NtQueryObject(ObjectNameInformation) on a wedged pipe handle never returns and the wait is not alertable. So the process never reaches main(), no git-level timeout can apply, no parent-side timeout can reach it once the parent is gone, and it survives indefinitely. Mine were killed manually at 6.8 hours.

The measurable signature, which I think is the useful part. Blocked before its first file read, such a process has done literally nothing, and the counters never move:

ProcessId KernelModeTime UserModeTime ReadOperationCount WriteOperationCount ageMin
    40788              0            0                  0                   0  425.3
    54996              0            0                  0                   0  425.3
    35868              0            0                  0                   0  425.3
     7180              0            0                  0                   0  425.2
    55596              0            0                  0                   0  425.2
    53560              0            0                  0                   0  425.2

24 DLLs were mapped and Ldr.Initialized: Yes, so the loader completed and it is genuinely parked in the kernel inside winansi_init(). For contrast, on the same host at the same moment, a real git reset --hard showed 209 reads / 8538 writes and another concurrent git showed 644 reads, within milliseconds of starting. Zero on all four is therefore a clean discriminator between "hung like this" and "slow but working", and it explains the "0 CPU-time delta" observations several people already reported in #17229.

Attribution. I ran a WMI __InstanceCreationEvent watcher over git.exe for 15 minutes, walking the parent chain of every spawn. 1230 git spawns were observed. Every process carrying this argv shape traced to codex.exe app-server (pid 35984) under ChatGPT.exe under explorer.exe. Zero came from any other agent on the box, and I had several running concurrently.

The six orphans' argv and cwd:

git.exe -c core.hooksPath=NUL -c core.fsmonitor=false status --porcelain     (x3)
git.exe -c core.hooksPath=NUL -c core.fsmonitor=false remote -v              (x1)
git.exe config --null --get core.fsmonitor                                   (x2)
CurrentDirectory: C:\tmp\cwt-chip-<redacted>-20260830\   (all six, a Codex worktree)

core.hooksPath=NUL matches DISABLED_HOOKS_PATH in codex-rs/git-utils/src/info.rs, and config --null --get core.fsmonitor matches codex-rs/git-utils/src/fsmonitor.rs and codex-rs/tui/src/get_git_diff.rs.

Why they end up orphaned, and the part I would like you to check.

On this host git resolves to C:\Program Files\Git\cmd\git.exe, a shim that spawns C:\Program Files\Git\mingw64\bin\git.exe as a child. So Codex's direct child is the shim and the process that hangs is a grandchild. I captured that two-level chain live:

pid=28584  "git" -c safe.bareRepository=explicit -c core.hooksPath=NUL -c core.fsmonitor=false remote get-url origin
    ^ pid=35984 codex.exe ... app-server
pid=37004  git.exe -c safe.bareRepository=explicit -c core.hooksPath=NUL -c core.fsmonitor=false remote get-url origin
    ^ pid=28584 (the shim)

Anything that reaps only the direct child leaves the hung grandchild behind. Being careful about what I have and have not verified:

  • codex-rs/git-utils/src/git_process.rs does already do the right thing: JobObject::create() plus spawn_contained, with KillGitProcessTreeOnDrop and a preserve_descendants() call on the success path. I am not claiming that path is broken.
  • But spawn_git_command's Err(_) fallback sets command.creation_flags(0) and spawns with job: None, and there is no #[cfg(windows)] impl Drop for KillGitProcessTreeOnDrop. In that state the only reaper left is kill_on_drop(true), which terminates the direct child only, which is exactly the shape I observed. A failed contained spawn silently degrades to leaking the tree.
  • Separately, the probes in codex-rs/tui/src/get_git_diff.rs do not go through git_process.rs at all. They build a WorkspaceCommand and run it through AppServerWorkspaceCommandRunner, i.e. the app-server exec path. My orphans' parent was the app-server, and two of the six are exactly that probe (config --null --get core.fsmonitor). I could not verify from source whether the app-server exec path job-contains on Windows, and that is the specific thing I am asking you to check.

I am deliberately not asserting which of those two is the cause. I first assumed "Codex kills the wrapper and not the tree", then read git_process.rs and found that framing wrong for that path, so I am reporting the evidence and the two candidate gaps rather than a conclusion.

What steps can reproduce the bug?

I do not have a deterministic trigger for the NtQueryObject wedge itself; it is a race on the pipe handle, and here it surfaced in bursts (six orphans inside one 5.1 second window, in two ticks about 5 seconds apart). The detection is reliable, so to confirm it on your own machines:

  1. Run Codex on Windows against a git repo for a while, ideally with parallel tasks. Other reporters in Codex Windows App keeps spawning git.exe status --porcelain=v1 -z and leaves orphan git.exe / conhost.exe processes #17229 see it worst with subagents.
  2. Enumerate hung git with the zero-work predicate:
$now = Get-Date
$all = Get-CimInstance Win32_Process
foreach ($p in ($all | Where-Object { $_.Name -eq 'git.exe' })) {
  $ageMin = [math]::Round(($now - $p.CreationDate).TotalMinutes, 1)
  $zero = ($p.KernelModeTime -eq 0) -and ($p.UserModeTime -eq 0) -and
          ($p.ReadOperationCount -eq 0) -and ($p.WriteOperationCount -eq 0)
  if ($zero -and $ageMin -gt 2) {
    '{0} ppid={1} {2}min {3}' -f $p.ProcessId, $p.ParentProcessId, $ageMin,
      ($p.CommandLine -replace '\s+', ' ')
  }
}
  1. Confirm the stack on any hit. This attaches non-invasively and leaves the process running, so it does not destroy the evidence:
"C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\cdb.exe" -p <pid> -pv -c "~*k; q"

ntdll!NtQueryObject should appear four frames above BaseThreadInitThunk.

What is the expected behavior?

A git probe Codex has already given up on should not outlive the session, and certainly not outlive the app. Concretely:

  1. Make Windows job containment cover every git spawn, including the app-server / WorkspaceCommand probe path, not just git-utils/src/git_process.rs. Because the real git is a grandchild of the cmd\git.exe shim on a stock Git for Windows install, a job object or an explicit tree kill is the only thing that actually reaps it.
  2. Make spawn_git_command's fallback fail closed. Today, JobObject::create() or spawn_contained failing downgrades to an uncontained spawn whose only cleanup is kill_on_drop, which cannot reach the grandchild. Failing the probe outright beats leaking a process that can never exit.
  3. Optional, and it would sidestep the git bug entirely: detect_msys_tty only runs for handles whose GetFileType is FILE_TYPE_PIPE. spawn_git_command already passes Stdio::null() for stdin, which is why fd 0 is safe; fd 1 and 2 are piped() and are what hang. For the small existence probes, capturing output via a temp file instead of a pipe would mean git never enters the hanging path. This is a workaround for a Git for Windows bug rather than a Codex bug, so treat it as a suggestion, not the ask.

Additional information

Related: #17229 (same symptom, open since April, 33 comments, no root cause posted; several comments there independently report "all recorded parent PIDs no longer existed" and "0 CPU-time delta", which this stack explains) and #29110.

Reproduced on this host twice, seven days apart: 2026-08-23 (4 orphans, 2.5 to 4 minutes old when found) and 2026-08-30/31 (6 orphans, 6.8 hours old). Both times on trivial commands that finish in well under a second when they run at all.

I mitigated locally with a reaper that kills git.exe only when all four counters are exactly zero, the process is past an age threshold, its parent is gone or its parent PID was recycled, and it has no live child. That last clause matters: a cmd\git.exe shim parked in WaitForSingleObject over a real working git also reads as zero-work, and killing it would orphan live work. I verified the gate by orphaning a genuine git fsck (parent killed, 98613 reads, well past the age threshold) and confirming it was spared while all six wedged processes were reaped.

Happy to run further instrumentation on this host if it helps; it reproduces here regularly.

Activity

  1. added
    bugSomething isn't working
    appIssues related to the Codex desktop app
    windows-osIssues related to Codex on Windows systems
    app-serverIssues involving app server protocol or interfaces
    tool-callsIssues related to tool calling
    on Aug 31, 2026
  2. Evan-Haug commented on Aug 31, 2026

    @Evan-Haug
    Author

    Two additions after tracing the producer, prompted by the source lead in #29110.

    1. The concrete producer of the exact command set I saw orphaned.

    TurnMetadata::fetch_workspace_git_metadata in codex-rs/core/src/turn_metadata.rs, called from spawn_git_enrichment_task:

    let (head_commit_hash, associated_remote_urls, has_changes) = tokio::join!(
        get_head_commit_hash(&self.cwd),
        get_git_remote_urls_assume_git_repo(&self.cwd),
        get_has_changes_in_repo(&self.cwd, repo_root),
    );

    That is rev-parse HEAD, remote -v, and the status path (with its config --null --get core.fsmonitor probe in front), fanned out concurrently per enrichment round. It accounts for all six of my orphans and for the command list in #29110.

    2. A third candidate gap: cancellation, not just timeout.

    I listed the Err(_) fallback and the app-server WorkspaceCommand path above. There is a third: cancel_git_enrichment_task calls task.abort(), which drops the in-flight futures and with them the KillGitProcessTreeOnDrop guard.

    That guard has an impl Drop only under #[cfg(unix)], where it calls kill_process_group. There is no #[cfg(windows)] impl Drop, so on Windows an aborted enrichment task relies entirely on the JobObject handle drop plus tokio's kill_on_drop(true) — and kill_on_drop reaps only the direct child, which on a stock Git for Windows install is the cmd\git.exe shim, not the mingw64\bin\git.exe grandchild that actually wedges.

    So the question is narrower than I first framed it: on Windows, does dropping the JobObject in KillGitProcessTreeOnDrop actually terminate the job (i.e. is JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE set on it), and does that hold on the abort path as well as the timeout path? If it does, the remaining exposure is the uncontained Err(_) fallback and any spawn path that does not go through spawn_git_command at all. I could not settle that from the published source, so I am leaving it as the ask rather than asserting a defect.

    Nothing here changes the git-side root cause in the original report; it only sharpens which Codex path lets the wedged grandchild survive.

  3. Evan-Haug commented on Aug 31, 2026

    @Evan-Haug
    Author

    Posting the local mitigation I described above, in case it is useful to anyone hitting this before the Codex-side containment question is settled:

    https://gist.github.com/Evan-Haug/b2020228cd4e1c02ea23b96b787375b2

    It is a workaround, not a fix. It does not patch Codex or Git, change any Git configuration, touch repositories, or need elevation. Report-only by default; -Apply is required to kill anything.

    The gate is the zero-work signature from the original report, plus three guards that turned out to matter:

    • All four of KernelModeTime, UserModeTime, ReadOperationCount, WriteOperationCount exactly 0. A process blocked before its first file read has done nothing that killing it can undo.
    • Parent provably gone — either absent from the process table, or present but created after the child, meaning the PID was recycled and is not the real parent. Without that second test a recycled PID reads as a live parent and the orphan becomes immortal.
    • No live child. This is the one I would flag to anyone writing their own version: a cmd\git.exe shim parked in WaitForSingleObject over a real, working git also reads as zero-work, and killing it would orphan the work it is waiting on.

    The false-positive case I tested explicitly, since it is the one that would hurt: a genuine git fsck with its parent killed out from under it, 98,613 reads, well past the age gate, was correctly spared while all six wedged processes were reaped. Long clone / fetch / worktree add all read continuously and are all spared for the same reason.

    Two caveats so nobody is surprised by the file:

    1. The script predates this investigation and carries two older, unrelated rules (orphaned git ls-files --others scans, and orphaned/runaway rg). Only the ZeroWork shape is about this issue. The README breaks that down.
    2. It also has an age-only rule for codex-windows-sandbox-setup.exe at a 30-minute cap, from a separate incident on this host: ten of them alive 21.9 hours having burned 244,436 CPU-seconds between them, with a live parent throughout, so the orphan rule could never have caught them. I have not investigated that one and it is not part of this report — flagging it only in case it is useful signal to you.

    Happy to strip it down to just the ZeroWork clause if a minimal reference version would be more useful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    appIssues related to the Codex desktop appapp-serverIssues involving app server protocol or interfacesbugSomething isn't workingtool-callsIssues related to tool callingwindows-osIssues related to Codex on Windows systems

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions