Watch mode: ~2.5-4 s of fixed setup per iteration in large monorepos (git status -uall, O(n²) queue scan, state hashes) #6122

🤖 Filed by GitHub Copilot acting for Naman Kanakiya (@namankanakiya) (light review only).

Repro context: a large private monorepo (~240,000 tracked files, ~1,040 operations in the watch selection), Rush 5.178.1, Node 22.16, Linux, 16 cores. rush start --to <tag> (also run through a wrapper that adds plugins), editing one source file of a library that ~1,000 operations depend on.

Observed: on every watch iteration, Rush's own performance marks (written to common/temp/telemetry) show 2.2-2.9 s (median about 2.35 s) between scheduling the iteration and rush:executionManager:configureIteration, before any operation starts. A CPU profile of the rush process (node --cpu-prof) over two edits attributes it to:

1. Inputs snapshot (getDetailedRepoStateAsync in package-deps-hash): git --no-optional-locks status -z -u --no-renames --ignore-submodules --no-ahead-behind (untracked files: all) takes 1.07 s wall on this repository. The same command with -unormal takes 0.52 s, because git's untracked cache (core.untrackedCache=true) only serves normal mode; with -uno it takes 0.18 s. git ls-files (0.17 s) runs in parallel and is not the bottleneck. Parsing the ls-files output costs 0.17 s CPU.
2. State hashes: OperationExecutionRecord.getStateHash for ~1,040 records costs 0.67 s CPU, mostly InputsSnapshot.getTrackedFileHashesForOperation (LookupByPath lookups over all tracked files). It is recomputed for every operation on every iteration, although almost no operation's tracked files changed.
3. AsyncOperationQueue.assignOperations: called from every next() and complete(), it scans the whole queue (for (let i = queue.length - 1; ...)) and compares each record's status against eight enum members, each read through a webpack harmony getter (_OperationStatus__WEBPACK_IMPORTED_MODULE_0__.OperationStatus.X). With ~1,040 records that is O(n²) per iteration: 1.05 s and 2.06 s CPU in the two profiled iterations. The profiler inflates this; the unprofiled cost is still to be confirmed.

Proposed changes (one PR each, or one PR with three commits)

(a) package-deps-hash: use -unormal and expand untracked directories

• Run git status -z -unormal ... instead of -u (all).
• For each entry that is an untracked directory (path ends with /), enumerate its files while respecting .gitignore (for example git ls-files -o --exclude-standard -z -- <dir>), and only for those directories.
• Expected: about 0.5 s less per iteration on this repository, for rush start, rush build (initial snapshot) and any other caller of getRepoStateAsync. The returned file list is identical.
• Test: a fixture repository with an untracked directory containing nested files; assert that the returned map matches the -uall result.

(b) AsyncOperationQueue: maintain a ready set

• Keep Ready records in a separate structure that complete() fills when a consumer's last dependency completes, and remove terminal records when they complete, so assignOperations() only looks at ready records.
• Hoist the status constants to module-level locals (or use a Set of terminal statuses) so the hot path does not go through harmony getters.
• Expected: removes the O(n²) scan, worth up to about 1 s CPU per iteration at ~1,000 operations.
• Test: the existing AsyncOperationQueue tests (ordering, and cobuild re-queue semantics: timesQueued ordering must be kept).

(c) Reuse per-operation state hashes across iterations

• Cache getOperationOwnStateHash per operation, keyed by the identity of the operation's tracked-file map, or skip recomputation when the new snapshot's hashes for the operation's project folder are unchanged (a per-project digest computed once per snapshot).
• Expected: most of the 0.67 s CPU per iteration.

Evidence

• Telemetry: common/temp/telemetry/telemetry_*.json, the performanceEntries of rush start iterations.
• A CPU profile of the rush main thread, aggregated over the first 6 s after each edit.
• git timings: date +%s%N around each command, three runs each, on the same checkout.

Happy to share more detail or test a candidate fix against this repository.