Skip to content

Make Rust CLI ownership crash-safe on Windows - #2458

Merged
SteveSandersonMS merged 5 commits into
mainfrom
gimenete-fix-cli-process-tree-ownership
Sep 1, 2026
Merged

Make Rust CLI ownership crash-safe on Windows#2458
SteveSandersonMS merged 5 commits into
mainfrom
gimenete-fix-cli-process-tree-ownership

Conversation

@gimenete

@gimenete gimenete commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Summary

  • spawn Rust SDK-owned CLI processes suspended on Windows, assign them to a private Job Object configured with JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE, and resume them only after assignment
  • retain the Job Object for the client lifetime so Windows terminates the owned CLI process tree even when the SDK-hosting process exits abruptly and Rust cleanup never runs
  • leave non-Windows process spawning and direct-child lifecycle behavior unchanged
  • add a Windows real-CLI E2E that terminates a separate SDK-hosting fixture with TerminateProcess and verifies the CLI exits through Job Object handle closure

Validation

  • cargo +nightly-2026-04-14 fmt --all -- --config-path .rustfmt.nightly.toml --check
  • cargo clippy --all-features --all-targets -- -D warnings
  • cargo test --lib --all-features
  • cargo check --target x86_64-pc-windows-msvc --lib --all-features

Related: github/app#2303

Contain SDK-spawned CLI descendants in Unix process groups and Windows Job Objects, and terminate the complete owned tree during shutdown and failure cleanup.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@gimenete
gimenete requested a review from a team as a code owner September 1, 2026 15:48
Copilot AI balanced review requested due to automatic review settings September 1, 2026 15:48

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Unix graceful shutdown can signal a reused process-group ID after reaping the root.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review tier: Balanced
Findings: 1 High severity

New issues introduced by this change (1)
Severity Finding
High severity rust/​src/​lib.rs — Consume the tree before reaping the root. On Unix, terminate() does not disarm ProcessTree, so…
What changed in this PR

Adds cross-platform lifecycle ownership for Rust SDK-spawned CLI process trees.

Changes:

  • Uses Unix process groups and Windows Job Objects.
  • Terminates descendants across stop, force-stop, drop, and startup failure.
  • Adds real-process regression tests.
File Description
rust/​src/​process_tree.rs Implements platform-specific process-tree containment.
rust/​src/​lib.rs Integrates containment into client lifecycle and tests.
rust/​Cargo.toml Adds platform-specific system dependencies.
rust/​Cargo.lock Locks the new dependencies.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread rust/src/lib.rs Outdated
Consume process-tree ownership during termination to prevent a second Unix signal after PID reuse, and make the Windows failed-start fixture execute through a PowerShell script.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@gimenete
gimenete requested a balanced review from Copilot September 1, 2026 17:14
@SteveSandersonMS

Copy link
Copy Markdown
Contributor

Thanks for continuing to tighten this. The process primitive itself is better than #2448, especially on Windows where the root is created suspended, assigned to the Job Object, and only then resumed.

However, I still don't see a real E2E proving the scenarios this PR is meant to address. #2448's real CLI validation showed the named cases did not hold up: shell tool children deliberately escape the CLI root process group, MCP servers already exit when stdio closes even without SDK process-tree termination, and subagents do not appear to be separate OS processes.

Can we lock in what specific real-world cases this is intended to address, and figure out what sort of E2E test would accurately represent them? My guess is there is some real-world GitHub App scenario that would be affected but we can't tell what that is from this PR, and we would need to be able to explain to other SDK consumers in which cases it deals with cleaning up the child processes it creates.

For tracking, I've moved this to draft — please mark as ready to review when appropriate.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Low-level cross-platform process containment requires final validation on native Windows and Unix environments.

Review tier: Balanced
Findings: None

Issues resolved since last review (1)
Severity Finding
High severity rust/​src/​lib.rs — Consume the tree before reaping the root. On Unix, terminate() does not disarm ProcessTree, so… View resolved comment

@SteveSandersonMS

Copy link
Copy Markdown
Contributor

I want to share an updated understanding of the problem here, since it changes what I think this PR should actually contain.

github/app#2303, the issue this PR references, is Windows-only, and on inspection it's a crash-safety bug, not a descendant-containment problem: the leaked processes have zero descendants. The CLI host is already killed directly by this SDK's existing stop/force_stop/Drop code today — the actual failure is that when the process hosting the SDK itself is killed or crashes, none of its own cleanup code (including this PR's Drop for ProcessTree) ever gets to run, so the CLI host is never told to die. On Windows, JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE fixes exactly this, because the OS itself tears down the job when the last handle closes, even if the owning process was killed abruptly with no code running. That part of this PR looks like the right, minimal primitive for the actual bug.

None of the four new tests exercise that property, though — they all go through the SDK's own graceful stop/force_stop/drop paths. I think we need a test that kills the process hosting the SDK itself (not calling stop/force_stop/drop) and confirms the CLI host still dies via the Job Object. I'm going to attempt adding this test myself.

Separately, I'd like to reduce this PR to the Windows-only path unless there's a known real-world scenario where the Unix process-group containment helps on Linux/macOS, demonstrated with its own E2E test. Without that, the Unix side isn't fixing anything we have evidence is broken, and it reintroduces the shell/MCP/subagent rationale that #2448's own investigation showed doesn't hold up, along with real risk (e.g., the PGID-reuse double-terminate issue already flagged on this PR).

@gimenete

gimenete commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

@SteveSandersonMS makes sense. I'll update the PR based on your comments 👍

SteveSandersonMS and others added 3 commits September 1, 2026 17:54
- Add stop_terminates_real_cli_wrapper_descendants: a Unix E2E test that
  launches the real bundled CLI through a shell wrapper (via
  ClientOptions::prefix_args) that backgrounds a descendant process, and
  verifies Client::stop() kills that inherited descendant.
- Add abrupt_host_termination_still_kills_cli_via_job_object: a Windows
  E2E test representing the actual github/app#2303 scenario -- an
  SDK-embedding host process being killed/crashing abruptly, so none of
  its own cleanup code (Client::stop/force_stop/Drop) ever runs. It
  spawns a new copilot-host-crash-fixture helper binary that starts a
  real CLI client and then never calls any SDK teardown code, terminates
  that helper abruptly, and asserts the CLI still dies via the Job
  Object's kill-on-close semantics.

Manually reproducing the same abrupt-kill scenario on Linux showed the
CLI already exits on its own within ~200ms once the OS closes the dead
host's end of the stdio pipe, with none of this crate's code involved --
a pre-existing, code-free safety net specific to stdio-piped processes
on Unix. That's further evidence Unix process-group containment isn't
needed to address #2303's failure mode, which is Windows-only.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use a kill-on-close Job Object only on Windows and test that abruptly terminating the SDK host tears down its owned CLI process. Preserve direct-child behavior on other platforms.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Keep the real-CLI abrupt-host E2E and remove unsupported Unix descendant coverage and the redundant synthetic crash test.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@gimenete gimenete changed the title Fix SDK-owned CLI process tree lifecycle Make Rust CLI ownership crash-safe on Windows Sep 1, 2026
@gimenete

gimenete commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

Addressed in 7576fdea. The PR is now scoped specifically to the Windows host-crash failure in github/app#2303: SDK-spawned roots are assigned to a kill-on-close Job Object before they can run, while non-Windows spawning remains unchanged.

The unsupported Unix process-group and shell/MCP/subagent rationale and tests are removed. The retained Windows E2E starts a real CLI from a separate SDK-hosting fixture, terminates that host with TerminateProcess so no Rust cleanup runs, and verifies Job Object handle closure terminates the CLI. In the app scenario the SDK host is the GitHub Copilot app process; Agency may be the SDK-spawned root and its Copilot CLI remains contained as a descendant.

@SteveSandersonMS
SteveSandersonMS marked this pull request as ready for review September 1, 2026 18:44

@SteveSandersonMS SteveSandersonMS left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review complete: scoped Windows crash-safety fix is correct and now covered by a real host-crash E2E.

@SteveSandersonMS
SteveSandersonMS added this pull request to the merge queue Sep 1, 2026
Merged via the queue into main with commit 76f6b6d Sep 1, 2026
52 checks passed
@SteveSandersonMS
SteveSandersonMS deleted the gimenete-fix-cli-process-tree-ownership branch September 1, 2026 19:20
dandriscoll added a commit that referenced this pull request Sep 1, 2026
Resolve the process-tree conflicts with #2458 (Windows crash-safe CLI
ownership), which landed on main after this branch's last merge. Keep the
PR's cross-platform whole-tree ownership (Unix process group + Windows Job
Object, active terminate through stop/force_stop/Drop, RAII startup guard)
and adopt main's Windows suspended-spawn + resume-after-assign that closes
the Job Object assign race. On containment-setup failure the child is still
resumed and teardown degrades to root-only rather than failing startup,
preserving the PR's reviewed soft-fallback.

- rust/Cargo.toml: union windows-sys features (add ToolHelp) and keep the
  unix libc dependency.
- rust/src/process_tree.rs: cross-platform module; Windows configure sets
  CREATE_SUSPENDED, attach assigns the Job Object then resumes the initial
  thread; test-only resume_without_containment lets the root-only teardown
  tests run a child configure spawned suspended.
- rust/src/lib.rs: thread extension_launch_provider through
  from_spawned_transport (main added it to from_transport); keep the
  guard-based spawn wiring and terminate routing.

Validation on Windows (COPILOT_SKIP_CLI_DOWNLOAD=1):
- cargo test --no-default-features --features test-support --lib: 230 passed,
  0 failed (including all 8 process-lifecycle tests).
- cargo +nightly-2026-04-14 fmt --all -- --config-path .rustfmt.nightly.toml
  --check: clean.
- cargo clippy (unwrap_used, disallowed_macros, await_holding_invalid_type):
  clean on the changed code.
- cargo doc -D warnings: clean.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants