✨ - Support added screenshot evidence in progress assessments or report exact unsupported binary paths #760

Is there an existing issue or feature request already for this?

• I have searched the existing issues and feature requests.

Related closed report: #727. This example concerns a new supplemental PNG screenshot in the actual PR delta, rather than inherited production binaries excluded by choosing the native PR base. It also relates to #726. Please consolidate if this capability belongs in the same workstream; the added-image reproducer and diagnostic requirements should remain covered.

What area does this improvement affect?

Other — bounded progress evidence and diagnostics.

What actions does this improvement affect?

Commit, Issue, Pull Request.

Description of the idea or improvement

Please support an explicitly bounded, exact-source representation of supplemental image evidence in progress assessments, or give a precise unsupported-binary diagnostic listing each affected path and a supported way to retain that evidence. Do not treat metadata or a screenshot as proof that product tests executed, and do not silently ignore arbitrary binaries.

A screenshot from actual application dogfooding was added alongside text source, executed test receipts, hashes and a verification report. GitHub returned no textual patch for that one PNG. The commit action failed with nonretryable validation.evidence-unavailable, retained progress and advised splitting the work, without identifying the affected path. All textual patches were available. The underlying fail-closed decision is understandable; the missing capability and generic recovery guidance make ordinary human-review evidence unnecessarily disruptive.

Current limitations or challenges

Concrete consumer: vypdev/vaultstadio PR #122, issue #78.

• Head: 5321a3b6596b2b73cb4a1cd6b5208c76d59edf2b.
• Development base: 15574b973688b06715cb263c518a3b6eaa599617.
• Failing run: https://github.com/vypdev/vaultstadio/actions/runs/40676625213, job 122776573964.
• Action: vypdev/copilot@v3, resolved in that job to 39679e26a22b40a5a4a40e8f167f46273157182e.
• Exactly one changed path lacks a patch: docs/development/verification/assets/transfer78-native-save.png, status added, additions/deletions/changes all zero. The original file is 244,037 bytes, SHA256 b715fdebce39f491419eee4a1295274d237fff05fa0ac5c95db0a23e29a9e962.
• The image shows the final native candidate after a real Save File action; separate original/saved bytes and hashes, source-bound tests and GUI receipts establish the functional claims. The image is supplemental, not a replacement for those checks.

Read-only reproduction:

gh api 'repos/vypdev/vaultstadio/compare/15574b973688b06715cb263c518a3b6eaa599617...5321a3b6596b2b73cb4a1cd6b5208c76d59edf2b' \
  --jq '[.files[] | select(.patch == null) | {filename,status,sha,additions,deletions,changes}]'

The PR-files endpoint independently returned only that same missing PNG patch at the observed head. The final workaround removes the image from the base-to-head text diff while linking its immutable published evidence commit; both the original local screenshot and the published image/history are preserved. No consumer Copilot code, generated workflows, scanner rules, protection or fabricated status is changed.

Relevant log:

GitHub omitted a required file patch from the pinned comparison. Progress was not assessed and existing state was retained.
Impact: Progress was not assessed because GitHub omitted a required file patch.
Error code: validation.evidence-unavailable
Action: Inspect the exact base/head comparison manually or split the work into smaller reviewed changes with complete patches before a new assessment. Retrying the same commits cannot restore omitted evidence.

The log also correctly skips push Bugbot analysis because PR #122 owns review for that exact head; that review-ownership skip is separate from the progress evidence failure. The temporary base switch used for issue linking is also separate and is not reported as a defect here.

Expected impact

Repositories can retain useful screenshots for human review without an opaque permanently red progress job, while evidence integrity remains explicit. If added images cannot be assessed, users receive the exact path, capability limit and supported recovery instead of having to identify omitted patches manually.

Suggested acceptance includes image add/modify/delete/rename, real exact-SHA content and byte limits, provider/truncation failures, binary-versus-missing-text distinctions, mixed text/image changes, stale heads, and preservation of existing progress when evidence is inadequate. Functional correctness must never be inferred from zero line counters, blob metadata or an image alone.

Alternatives considered

Keep screenshot evidence in an immutable published evidence commit and link it from the text report, leaving the final comparison text-only. This is the applied consumer workaround, with no evidence loss. It should be documented if it remains the recommended supported approach.

A blanket binary ignore or metadata-only success is not requested. The broader mechanism may require type-aware evidence and an explicit unsupported result rather than image analysis; please choose the design that preserves the verification contract.

Version of copilot

vypdev/copilot@v3, exact resolved source 39679e26a22b40a5a4a40e8f167f46273157182e in the linked run.

Additional context or comments

The local clean consumer implementation passed 6,355 product executions and actual Web/macOS byte/hash journeys; final-ref CI remains independently required. Those successful tests do not make missing evidence acceptable. This feature request concerns how supplemental nontext evidence is represented and diagnosed, rather than weakening the assessment or requesting automatic approval.