[rush] Rewriting a tracked input file with identical content blocks build cache writes and breaks cobuild #6149

Summary

• Rewriting a tracked input file with identical bytes makes Rush skip the build cache write for that operation and for every operation downstream of it
• With cobuild, each runner then executes every cobuilt operation itself
• Seen on 5.181.0, not on 5.169.3

Example

Two projects, gen and app (app depends on gen, and has test shards run through cobuild).

gen/package.json:

{ "scripts": { "_phase:build": "node write-generated.js" } }

gen/write-generated.js (src/generated.txt is committed to git):

require('fs').writeFileSync('src/generated.txt', 'same content\n');

Build cache with writes allowed (S3), cobuild plugin, 3 runners.

What happens

• gen (build) logs Input files changed after the inputs snapshot was taken; not writing a build cache entry.
• Only the mtime changes. Inode and size are the same, the bytes are the same
• No cache entry for gen, and none for the app operations that depend on it
• All 3 runners run all 10 app test shards. The job hit its 20 minute limit (cancelled at 20m18s). On 5.169.3 it takes about 10 minutes
• rush --debug test -v shows Failed to acquire lock ... locked by runner shard-N repeating while the runners wait on each other

Workaround

• The script now writes only when the content differs
• Result on 5.181.0: the message appears 0 times on all 3 runners, the job takes 10m29s to 11m02s, and each shard runs once

Cause (from reading the source)

• The check compares a stat signature (size, mtime, inode) of the tracked input files from before and after the operation. Content is never compared
• Consumers' cache keys embed the producer's pre-edit state, so their writes are blocked too. That explains why dependents lose caching

Not verified

• Not reproduced locally: macOS, local-only cache, rush rebuild, same rewrite, the message never printed
• Only seen on Linux CI with an S3 cache and cobuild
• Not the cobuild plugin version: it fails the same on 5.176.x and 5.181.x

Ask

1. Hash the content when the stat signature differs, and treat identical content as unchanged
2. Name the changed file(s) in the log message. Finding the cause meant reading Rush's source

Is the stat-only comparison intentional, or is a content check welcome?

Standard questions