Skip to content

[rush] Build cache output receipts are invalidated by ctime/mode-only changes, so every operation is restored again聽#6128

Open

Description

馃 Filed by GitHub Copilot acting for Naman Kanakiya (@namankanakiya) (light review only).

Summary: the build cache's output-folder receipts are invalidated by changes that touch only file metadata (ctime, or mode bits), so every cacheable operation is restored from the cache again even though its outputs are already on disk and unchanged.

Repro:

  1. rush build with the build cache enabled; outputs are restored or built and receipts are written.
  2. Change only metadata on the repo's files without touching content, size, inode or mtime. For example chmod -R g+w ., setfacl -R -m u:<user>:rwX ., or anything that rewrites ctime (copying a snapshot of the volume that preserves mtime but not ctime).
  3. rush build again.

Actual: every cacheable operation is restored from the cache again.

Expected: the operations are certified as up to date and skipped, because their outputs are identical.

Cause: the receipt's listing digest includes each output file's ctime and full mode (alongside dev, ino, size, mtime). Any ctime change, which the kernel applies on chmod/chown/ACL changes and on many snapshot/restore paths, mismatches every receipt.

Impact: cloud dev environments created from prebuilt snapshots (where outputs were already restored during the prebuild) re-restore every operation on the first build after start. We observed 758 operations and 16-20 minutes, much of it reading cache tarballs from a lazily hydrated disk. In that environment, sampled output files had ctime = mtime + 470 s exactly, with unchanged mode and content.

Suggestion (either):

  • Certify by path, ino, size and mtime (plus the executable bits) only, excluding ctime and non-exec mode bits; or
  • On a listing digest mismatch where path, ino, size and mtime are identical, verify the outputs' content hashes against the cache entry before extracting again.

Happy to test a candidate fix.

Activity

  1. namankanakiya commented on Oct 13, 2026

    @namankanakiya
    Author

    馃 Comment by GitHub Copilot acting for Naman Kanakiya (@namankanakiya) (light review only).

    More data from the prebuilt-snapshot environment described above. Mode bits, ACLs and xattrs are not involved there. The only things that change are file ctime and output-folder mtime:

    • A step that runs after the prebuild's cache restore, but before the snapshot is taken, moved ctime on about 85% of the output files, and mtime on most output root folders.
    • File content, size, inode, file mtime, mode, owner, default ACL and xattrs were all unchanged. 31,769 output files across 50 random operations hash identical to their cache entries.
    • The snapshot copy itself preserves inode, mtime and ctime exactly.

    So a pure ctime change is enough to invalidate the receipts; no chmod/setfacl is needed. Any fingerprint that includes ctime, or directory mtime, is fragile across this kind of environment. Inode plus content-defining fields (size, file mtime) survive it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    • Status
      Needs triage

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions