[Bug]: Non-sparse skip-worktree entries still force full checkpoint index rebuild after #10792 / #12154 #13741

Before submitting

• Searched existing issues. Related reports and merged fixes are linked below; this report concerns the remaining non-sparse checkout with special index flags path.
• Included the observed environment, triggering conditions, implementation analysis, and local validation results.

Area

apps/server — Git checkpoint capture

Version / environment

• Windows desktop, 0.0.43-nightly.20260925.2269 (also observed on 0.0.43-nightly.20260923.2173).
• Windows 11; Git for Windows 2.54.0.windows.1.
• Worktree accessed through a mapped SMB drive.
• Non-sparse checkout, approximately 24,804 tracked index entries, with 6 manually skip-worktree-marked entries.

Related fixes / reports

• fix(server): reuse Git index metadata during checkpoint capture #10792 — merged index-metadata reuse optimization.
• fix(server): keep large sparse checkouts on the fast checkpoint path #12154 — merged fix for large sparse checkouts; this is a follow-up for the non-sparse special-flags branch, not the sparse-output-limit case.
• Checkpoint index reuse falls back on large sparse checkouts and still fails after #10792 #12080, [Bug]: Checkpoint capture on large monorepos retries a guaranteed 30s git-add timeout every turn — permanent CPU burn + tmp_pack disk litter #3646, captureCheckpoint re-hashes the whole worktree on every checkpoint, so large repos always hit the 30s VCS timeout #12924 — related checkpoint fallback / whole-worktree hashing reports.

Steps / triggering conditions

1. Open an existing non-sparse repository with some tracked entries marked skip-worktree in the Windows desktop app. A sufficiently large worktree or higher-latency filesystem makes the cost visible.
2. Start a turn that captures a checkpoint.
3. Observe checkpoint capture repeatedly hit the 30-second VCS timeout, even though ordinary status/diff queries complete within the limit.

This is a repeatedly observed real-worktree reproduction, not a claim that every small repository with a skip-worktree entry will time out. The affected implementation also rejects assume-unchanged flags; the real-worktree timeout observation here involved skip-worktree entries.

Expected behavior

Special flags on a handful of entries should not force loss of cached metadata for every other tracked file on each checkpoint. The checkpoint must still capture changes/deletions in flagged files correctly, without changing the user's real index or configuration.

Actual behavior / impact

Major degradation or frequent failure: checkpoints repeatedly fail.

VCS process timed out in GitVcsDriver.checkpoints.captureCheckpoint:
git (<redacted-worktree>) after 30000ms

Read-only baseline timings in the affected worktree:

• git --no-optional-locks status --porcelain=v1 -uall: 9.84 s.
• git --no-optional-locks diff --stat: 5.84 s.
• git ls-files -v -z: 0.32 s.

Historical logs contained 16 matching checkpoint timeout spans. Inspected traces consistently showed seven successful Git calls within capture followed by an approximately 30-second failing call. That order matches the fallback path below and its subsequent add -A; existing traces do not include argv, so the individual failing subcommand is inferred rather than directly recorded.

Implementation analysis

Inspection of the installed build shows:

1. Copy the real index to a private GIT_INDEX_FILE and run read-tree --reset HEAD.
2. Inspect ls-files -v -z flags.
3. Any lowercase flag, or S when !sparseCheckout, sets specialFlags = true.
4. Reject the copied index, remove it, and seed a fresh index with read-tree HEAD.
5. Run git add -A -- . with the default 30-second command timeout.

Thus a few manually skipped entries still take the fresh-index path after the sparse-checkout fixes. Cached stat information for unrelated entries is discarded on every capture. Filesystem latency can amplify the work, but the repeatable trigger is this checkpoint-specific fallback.

Locally validated mitigation (not an upstream PR yet)

A local experimental patch retains the copied index for the non-sparse branch and rebuilds only the flagged entries in the private index:

• Collect mode, OID, stage, and NUL-delimited paths via ls-files --stage -v -z.
• Feed those entries to update-index -z --index-info with the private GIT_INDEX_FILE, preserving tracked status and OIDs while clearing those entries' flags/stat cache.
• Restore the existing conservative index timestamp; retain the existing safe fallback on failure.
• Leave sparse-checkout handling and unflagged entries' metadata unchanged.

Simply clearing flags was not sufficient in local verification: same-size/same-mtime content edits could still reuse stale metadata. Removing entries outright would also need care for ignored-but-tracked files. The selective index-info approach avoids those two pitfalls.

Isolated checks covered h/S/s changes and deletions, same-size/same-mtime edits, racy timestamps, ignored-but-tracked files, staged/worktree differences, root/subdirectory scope, and unchanged source-index hashes. Snapshot trees matched the clean-index baseline; 62 unflagged entries retained their stat/OID metadata. Limitations: the isolated harness substituted a tree OID for HEAD, symlink representation used Windows core.symlinks=false, and actual submodule HEAD changes were not exercised.

After applying the local patch and fully restarting T3, one real checkpoint completed successfully in 11.09 s, with its outer event chain also succeeding. This is one successful post-patch runtime observation, not a long-running benchmark or a claim of complete correctness coverage. No user Git configuration or real-index flags were changed to obtain this result.

Would selectively rebuilding special-flag entries in the private index be an acceptable upstream direction? More granular checkpoint subcommand diagnostics would also make this failure much easier to investigate.