Skip to content

fix(checks): make optional evidence failures actionable - #753

Merged
efraespada merged 1 commit into
developfrom
bugfix/728-optional-check-run-publication-fails-without-actionable-diagnostics-in-the-commit-workflow
Oct 28, 2026
Merged

efraespada merged 1 commit into
developfrom
bugfix/728-optional-check-run-publication-fails-without-actionable-diagnostics-in-the-commit-workflow

Conversation

@efraespada

@efraespada efraespada commented Oct 28, 2026 •

Copy link
Copy Markdown
Member

Optional Check Run publication previously lost the failed operation and provider facts, and a custom semantic error could leak its private message. This change reports a bounded, localized diagnostic in the run log and a Job Summary appendix while preserving required-stage success/failure and completed effects.

What changed

  • Close SDK failures at the evidence adapter boundary with a classified cause, validated HTTP status and safe request reference. Keep the existing check name/head/conclusion, validate the returned native ID and make no automatic write retry.
  • Show the intended checks.create operation, complete valid head, check name and credential-selection route. Unknown failures stay unknown; credentials, raw bodies/headers, URLs and exception messages are excluded.
  • Supply the step-scoped native Actions token and exact Checks-write/contents-read permissions in installed and repository commit workflows. Product operations keep their configured token. Structural tests reject missing, misplaced or overbroad alternatives.
  • Update the owning SDD/catalog, numeric and exhaustive coverage budgets, recovery documentation and generated bundles. Existing consumers adopt the workflow/runtime through their normal reviewed upgrade; no consumer change or live outage/permission cause is claimed.

Validation

  • Full coverage: 756 root Jest suites, 13,433 passing tests and 35 existing skips; all global/module and setup-acceptance budgets passed. Extension: 267 tests and 100% in all four metrics.
  • New closed error, classifier, presenter and evidence adapter: 100% in all metrics. Changed-line audit: 48 covered executable lines, zero misses/uncovered branches. Remote Codecov remains an independent gate.
  • 455 directed tests, complete type/lint/spec/docs/workflow checks, generated bundles, npm package and CLI + typed API + isolated local-web smoke passed. Graphify updated.
  • Before/after proof: original completion leaked a custom semantic error message; the privacy regression fails before and passes after. The raw-error guard recognizes only the actual production formatter symbol and rejects a same-named local spoof. No catch/file exclusion or threshold reduction.

Review notes

The original consumer warnings establish no provider HTTP status, credential type or outage. A transported write can be completed before its response is lost, so recovery instructs inspection of the same native head before replay. Native Check identity and source eligibility are preserved; this does not add a provider-idempotent upsert registry or fabricate a head for an ineligible event. Optional publication and its Summary appendix never replace an earlier required error or turn successful required work into failure.

Closes #728

…agnostics-in-the-commit-workflow: retain safe optional-publication diagnostics
@docs-page

docs-page Bot commented Oct 28, 2026

Copy link
Copy Markdown

To preview the documentation for this pull request, visit the following URL:

docs.page/vypdev/copilot~753

Documentation is deployed and generated using docs.page

@vypbot
vypbot self-requested a review October 28, 2026 00:10
@vypbot vypbot added this to vypdev Oct 28, 2026
@vypbot vypbot moved this to In Progress in vypdev Oct 28, 2026
@vypbot
vypbot changed the base branch from develop to master October 28, 2026 00:10
@vypbot
vypbot changed the base branch from master to develop October 28, 2026 00:11
@vypbot vypbot added size: XXL Indicates an extremely large task or issue that requires significant time and effort to complete. 75% Progress: 75% labels Oct 28, 2026
@vypbot

vypbot commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

Bugbot: review complete

Current status: No active findings on 842d397.

Pull request · Verified commit · Workflow run

@vypbot

vypbot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Copilot approval · Human review needed

Status: A protected workflow, policy, or trust-boundary file changed.
Completed: 1/1 observed checks succeeded; Bugbot complete.
Action required: Ask a human reviewer to decide.
Impact: No new approval is confirmed for this revision.
See checks

Technical evidence

Revision: 842d397e266e; base: a953df8802ed; reason: protected-path.

@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.69%. Comparing base (5b3664f) to head (842d397).
⚠️ Report is 25 commits behind head on develop.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop     #753      +/-   ##
===========================================
+ Coverage    94.50%   94.69%   +0.19%     
===========================================
  Files         1035     1063      +28     
  Lines        33299    34239     +940     
  Branches      9824    10135     +311     
===========================================
+ Hits         31468    32423     +955     
+ Misses         699      684      -15     
  Partials      1132     1132              
Files with missing lines Coverage Δ
src/actions/github_action_completion.ts 93.47% <100.00%> (+0.56%) ⬆️
...ication/policies/action_summary_message_catalog.ts 100.00% <ø> (ø)
...ation/policies/optional_check_diagnostic_policy.ts 100.00% <100.00%> (ø)
src/application/ports/copilot_evidence_error.ts 100.00% <100.00%> (ø)
src/data/repository/copilot_evidence_repository.ts 100.00% <100.00%> (ø)
...sitory/github/copilot_evidence_provider_failure.ts 100.00% <100.00%> (ø)

... and 11 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@efraespada
efraespada merged commit 5269ef8 into develop Oct 28, 2026
13 checks passed
@github-project-automation github-project-automation Bot moved this from In Progress to Done in vypdev Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

75% Progress: 75% size: XXL Indicates an extremely large task or issue that requires significant time and effort to complete. state:verified

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

🐛🧑‍💻 - Optional Check Run publication fails without actionable diagnostics in the commit workflow

3 participants