[rush] Watch mode: an iteration whose inputs snapshot is older can replace (and abort) one whose snapshot is newer #6142

Summary

In watch mode, OperationGraph.#scheduleIterationAsync awaits the iteration's inputs snapshot before it schedules the iteration. Two triggers that arrive close together (for example a plugin's run request, then a file change) take their snapshots concurrently. Whichever snapshot completes last calls #setScheduledIteration and aborts the running iteration, even when its snapshot is older.

The older iteration then runs with inputs that predate the file change. The changed operations look up to date, so the change is not built by that iteration.

Versions

• @microsoft/rush 5.181.0 (also current main)
• Node.js 22, Linux

Repro

1. Large repo; rush start (watch) with a plugin that calls invalidateOperations() / requests a run when the watch goes idle (any run request works).
2. Right after the run request is queued (Run requested by ... Queuing new iteration...), save a source file of a watched project (File change detected. Queuing new iteration...).
3. Have the first snapshot take longer than the second. This happens naturally with a cold file system cache (snapshots of 5-60 s), or deterministically by wrapping getInputsSnapshotAsync to delay its first result.

Observed:

t+0.0s  Watch Status: Run requested by A (build) [...]. Queuing new iteration...   <- snapshot S1 starts
t+3.1s  Watch Status: File change detected. Queuing new iteration...               <- snapshot S2 starts
t+31.8s Executing ...            <- S2 completed first; its iteration (with the change) starts
t+35.7s ==[ ABORTED: 3 operations ]==   <- S1 completed; #setScheduledIteration replaced the plan, aborted S2's iteration
t+36.3s Executing ...            <- S1's iteration runs; the changed project is "up to date" by S1's hashes

The change is built only if something else later notices that the working tree differs from S1, and queues another iteration.

Expected

A snapshot that started before the snapshot of an iteration already scheduled or running never replaces or aborts that iteration. The newer iteration was planned later, from newer state, so it already covers the older trigger's work (and ProjectWatcher requeues any run request that no iteration served).

Actual

The last snapshot to complete wins, regardless of age. The newer iteration is aborted, and a stale iteration runs.

Code pointers (main)

• libraries/rush-lib/src/logic/operations/OperationGraph.ts
  • #L955-L963: startTime = performance.now() and inputsSnapshot = await getInputsSnapshotAsync?.(), before scheduling.
  • #L1101: this.#setScheduledIteration(iterationContext) unconditionally.
  • #L1112-L1116: void this.abortCurrentIterationAsync() when an iteration is running.
• libraries/rush-lib/src/logic/ProjectWatcher.ts
  • #L431-L447: each debounced trigger calls graph.scheduleIterationAsync({}) and tracks it in #pendingSnapshots, so several snapshots can be in flight at once.

Suggested fix

In #scheduleIterationAsync, record the startTime of the newest snapshot that has been scheduled. After the snapshot completes, if its startTime is older than that, do not call #setScheduledIteration and do not abort the current iteration; finalize its records as NoOp, the way the totalOperations === 0 path does. Alternatives: an epoch counter checked after await getInputsSnapshotAsync(), or serializing snapshot creation.

Workaround as a plugin (works today): a configureIteration tap at the last stage disables every record of an iteration whose iterationOptions.startTime is older than the newest one already configured (skipping calls with startedOperations). Rush then takes the totalOperations === 0 path and neither schedules nor aborts anything. A plugin that enables operations in configureIteration must then keep them until an iteration that enables them actually starts, since a later tap can discard the plan.

Filed with help from GitHub Copilot (AI); reviewed by the reporter.