🐛🧑‍💻 - Progress assessment fails permanently on inherited binary files without GitHub textual patches #727

Is there an existing issue for this?

• I have searched the existing issues.

Related: #726, #547 and #611. This report isolates ordinary binary files with intentionally unavailable textual patches, rather than the large textual comparison in #726. There are only 55 changed files and no 300-file boundary. Please consolidate with #726 if the maintainers prefer one evidence-recovery workstream; the reproducer and binary-specific acceptance cases should remain covered.

Which actions are affected?

Commits

Which platforms are affected?

Linux

Description

CheckProgressUseCase fails the consumer's Copilot - Commit job with nonretryable validation.evidence-unavailable after a valid merge from production includes five compiled CLI libraries. GitHub correctly omits textual patches for these binary files. Copilot requires a nonempty patch for every changed path except build/** and specs/CATALOG.md, so any such binary outside those special cases makes the complete progress assessment impossible.

The affected consumer is public repository vypdev/zstandard, maintenance PR vypdev/zstandard#172, issue vypdev/zstandard#171. This maintenance PR prepares the project's supported verification dependency hook before an authorized patch release. All five binary files were inherited unchanged from the existing production commit by a separately authorized one-time GitHub branch update; the maintenance PR's actual delta against master contains 15 text files and no binary modifications.

Expected: provide a supported, bounded, exact-SHA way to account for binary changes and production lineage without requiring nonexistent textual patches or telling consumers to split already-produced platform libraries. Do not infer functional correctness from file metadata or silently exclude arbitrary binaries. If a binary cannot be assessed, explain the exact paths and evidence requirements and distinguish unavailable progress from project verification failure. The correct policy belongs in Copilot; we have not patched the consumer's Core workflows, ignored the files, changed approval policy or synthesized a passing status.

Actual: no model assessment takes place; existing progress is correctly retained, but the push workflow is red and the generic recovery advises splitting the work. A retry of the identical immutable comparison cannot create binary patches, and adding an unrelated text commit cannot remove the binary files from the development-to-head comparison. Splitting the maintenance code also does not remove the inherited production binaries.

Reproducing the issue

1. Use vypdev/copilot@v3 as resolved in the failing job (exact version below).

2. Pin repository vypdev/zstandard, development base 22eb4de4ce8f5d959b735cb4c7a91c3ca35cf73e and source head 4ac3c633f45166acd2b14c169fe675b6e640d1a6. The head is a merge with parents fe974b5958ebfda902c870b40b9fd77df9c6ae48 and production 804c1a879ba54d766547ef31cb1ae8a88cfde8dc; its tree is 5de1d6cdf51bd39b0b8cd3521a846c4c4dfe663a.

3. Request the exact endpoint used by ProgressDiffRepository:

   gh api 'repos/vypdev/zstandard/compare/22eb4de4ce8f5d959b735cb4c7a91c3ca35cf73e...4ac3c633f45166acd2b14c169fe675b6e640d1a6?per_page=1&page=1' \
     --jq '{status, ahead_by, behind_by, total_commits, files: (.files | length), missing: [.files[] | select(.patch == null) | {filename,status,sha,additions,deletions,changes}]}'

   Actual: status=ahead, ahead_by=21, behind_by=0, total_commits=21, files=55. Exactly five files lack patches; all five report status=modified, additions=0, deletions=0, changes=0 despite different binary contents relative to development. These zero counters must not be interpreted as unchanged files.

   Path	Exact head Git blob, also identical in production 804c1a8
   zstandard_cli/lib/src/bin/libzstandard_linux_arm64.so	4de3c1c30eb5fce504c0fedfd39b0a8e91e8fa6d
   zstandard_cli/lib/src/bin/libzstandard_linux_x64.so	9ad2d1c1aff9c20f68157bbc4827572ca95dfc02
   zstandard_cli/lib/src/bin/libzstandard_macos.dylib	6cfb1e6caba56adf85171c895203d0ac72466d2c
   zstandard_cli/lib/src/bin/zstandard_windows_arm64.dll	26edc33f2550d8af891e3a99a3920ac2f8ad71be
   zstandard_cli/lib/src/bin/zstandard_windows_x64.dll	609cf719aacc04425418fa3fd1d8e637a6543db6

4. Independently fetch the public refs in an existing clone and check lineage/content:

   git show --format=fuller --no-patch 4ac3c633f45166acd2b14c169fe675b6e640d1a6
   git diff --exit-code 804c1a879ba54d766547ef31cb1ae8a88cfde8dc \
     4ac3c633f45166acd2b14c169fe675b6e640d1a6 -- zstandard_cli/lib/src/bin
   git diff --name-status 804c1a879ba54d766547ef31cb1ae8a88cfde8dc \
     4ac3c633f45166acd2b14c169fe675b6e640d1a6

   First diff is empty/exit 0; second lists the 15 project-maintenance text files. Those facts establish unchanged production inheritance, not functional correctness of the binary payloads.

5. Observe failing push run https://github.com/vypdev/zstandard/actions/runs/40338528076, job https://github.com/vypdev/zstandard/actions/runs/40338528076/job/121626513660, UTC 2026-10-26 14:20:21–14:20:30.

6. analyzeProgress pins the configured development branch head, rather than the actual maintenance PR base master; ProgressDiffRepository.getDiff rejects the first binary without a patch before model invocation. This is why the binaries remain relevant even though the maintenance PR itself does not change them.

copilot Version

3.10.5. vypdev/copilot@v3 resolved to 39679e26a22b40a5a4a40e8f167f46273157182e in the failing job; the repository uses a version tag as its Action reference. The exact commit is diagnostic evidence only.

Relevant source at the executed revision:

• ProgressDiffRepository: allows generated metadata only for build/ and specs/CATALOG.md; requires file.patch otherwise.
• analyzeProgress: comparison pins the configured development branch and current source SHA.

Relevant Log Output

2026-10-26T14:20:21.7654789Z Executing CheckProgressUseCase.
2026-10-26T14:20:23.8783527Z GitHub omitted a required file patch from the pinned comparison. Progress was not assessed and existing state was retained.
2026-10-26T14:20:24.3212587Z Skipping push Bugbot analysis because pull request #172 owns review for this head.
2026-10-26T14:20:30.2723945Z Could not publish the optional GitHub Check Run.
2026-10-26T14:20:30.2757783Z Impact: Progress was not assessed because GitHub omitted a required file patch.
Error code: validation.evidence-unavailable
Action: Inspect the exact base/head comparison manually or split the work into smaller reviewed changes with complete patches before a new assessment. Retrying the same commits cannot restore omitted evidence.
Retained state: Existing progress labels and published assessments were preserved; no model assessment was made.
Retryable: No
Reference: 8ee13d41-3952-4c5b-8bda-6bc8ff9672d4

The optional Check Run warning is an additional observation in this same run; no HTTP status or causal detail was present in the nearby log. We have not established whether it is related. The substantive failure above is independently reproducible from the immutable compare response.

Additional context and comments

The maintainer explicitly requested reporting all anomalous Copilot behavior upstream and leaving Copilot intact in the consumer. This issue reports the problem for the Copilot maintainers to diagnose and implement; it does not request an automated fix in zstandard or authorization to change its policies.

Evidence separating progress from verification:

• Fresh actual project setup and unit verification on this same head passed: https://github.com/vypdev/zstandard/actions/runs/40338540776/job/121626560008. The hook setup completed in 18 seconds, actual complete unit execution/source guard passed, and the owned profile was retired. Remaining platform CI was still running when this report was filed; no claim of a fully green run is made.
• Copilot's normal synchronization command confirms the branch already contains master: 🔧🧑‍💻 - Prepare project Flutter dependencies inside Copilot verification jobs zstandard#171 (comment). This already-current receipt does not prove a new SDK-dependent merge verification ran.
• Manual evidence review is recorded at ci: prepare exact project checks inside Copilot verification jobs zstandard#172 (comment). The failed progress status remains visible; no synthetic success or fake assessment was posted.
• The one-time bootstrap synchronization has explicit maintainer approval and a reconciliation record: 🔧🧑‍💻 - Prepare project Flutter dependencies inside Copilot verification jobs zstandard#171 (comment). Future managed synchronization remains Action-owned.
• Copilot engine, generated Copilot workflows, profile, approval policy, credentials and normal deployment lifecycle were not modified to work around this failure. The project-owned dependency hook is the documented extension point and remains disabled pending trusted-main review.

Suggested coverage for maintainers: binary add/modify/delete/rename; intentionally absent binary patches versus truncated text patches; zero-statistics entries with changed blobs; unchanged production inheritance into a candidate with a different PR base; bounded immutable blob/lineage evidence and stale-head detection; provider errors/resource limits; no model score or completion without adequate evidence; diagnostics that identify every unassessed path and offer a viable recovery. A binary evidence policy must not silently equate metadata or a successful compile with content review.