Skip to content

fix(bugbot): ground Action input suggestions in versioned contracts - #758

Merged
efraespada merged 1 commit into
developfrom
bugfix/739-bugbot-recommends-an-undeclared-setup-dart-input-despite-the-versioned-upstream-contract
Oct 28, 2026
Merged

efraespada merged 1 commit into
developfrom
bugfix/739-bugbot-recommends-an-undeclared-setup-dart-input-despite-the-versioned-upstream-contract

Conversation

@efraespada

@efraespada efraespada commented Oct 28, 2026 •

Copy link
Copy Markdown
Member

Bugbot could recommend replacing a valid external Action input with an undeclared key based on another version or model memory. Review and the real-agent benchmark now share a compact instruction that checks the exact Action revision, manifest/runtime getter, forwarded inputs, defaults and effective values before reporting a defect or suggesting a replacement. Unverifiable claims abstain; verified wrong-key and ignored-value failures remain reportable.

Closes #739.

Fifteen new delivery, verdict-isolation, scorer and native YAML/compiled-condition controls extend the corpus to 71 cases. The owning SDD, catalog, detection/quality documentation and runtime bundles are updated. The original long-ignore-list prompt-size limit remains enforced; the final instruction is 713 characters.

Validation:

  • 772 suites / 14,132 tests pass, 35 existing skips; all global/module coverage budgets and setup acceptance; 267 extension tests with 100% in all four metrics.
  • 91 directed tests, including the original prompt-size regression; types, lint, specifications/catalog, docs/workflows/agent docs, Graphify, build/package 87 files and local CLI/API/web smoke pass.
  • Local patch audit: 2 executable modified production lines covered; zero misses or uncovered changed branches. Normal commit hooks pass build/full Jest/lint.

Review evidence: the immutable upstream manifest declares problem-matcher; its runtime reads that input and disables registration for false. This is read-only source corroboration, separate from the consumer's reported SDK/production runs.

The actual measurements retain all four instruction variants, fixed inputs, identities, predictions and scores. The final sample reports all three real defects and no negative-case findings, but its full default score fails because two anchors are off by one and a category differs from the reference. The full-text intermediate passed model scoring but exceeded the existing context-size test; a compact intermediate missed the ignored-value case. Inputs, expectations and scoring thresholds are unchanged. Reference predictions validate the deterministic scorer; these optional measurements are not an all-thresholds-passed or statistical improvement claim. Model guidance remains probabilistic.

…he-versioned-upstream-contract: ground Action input suggestions in versioned contracts
@docs-page

docs-page Bot commented Oct 28, 2026

Copy link
Copy Markdown

To preview the documentation for this pull request, visit the following URL:

docs.page/vypdev/copilot~758

Documentation is deployed and generated using docs.page

@vypbot vypbot added state:ai-processing size: XL Indicates that the issue or task is extra large in scope and complexity. and removed state:ai-processing labels Oct 28, 2026
@vypbot
vypbot self-requested a review October 28, 2026 10:36
@vypbot vypbot added this to vypdev Oct 28, 2026
@vypbot vypbot moved this to In Progress in vypdev Oct 28, 2026
@vypbot
vypbot changed the base branch from develop to master October 28, 2026 10:36
@vypbot
vypbot changed the base branch from master to develop October 28, 2026 10:36
@vypbot

vypbot commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

Bugbot: review complete

Current status: No active findings on cafdeb5.

Pull request · Verified commit · Workflow run

@vypbot

vypbot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Copilot approval · Human review needed

Status: A protected workflow, policy, or trust-boundary file changed.
Completed: 1/1 observed checks succeeded; Bugbot complete.
Action required: Ask a human reviewer to decide.
Impact: No new approval is confirmed for this revision.
See checks

Technical evidence

Revision: cafdeb518482; base: e45011390516; reason: protected-path.

@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.79%. Comparing base (5b3664f) to head (cafdeb5).
⚠️ Report is 39 commits behind head on develop.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop     #758      +/-   ##
===========================================
+ Coverage    94.50%   94.79%   +0.28%     
===========================================
  Files         1035     1078      +43     
  Lines        33299    34859    +1560     
  Branches      9824    10354     +530     
===========================================
+ Hits         31468    33043    +1575     
+ Misses         699      684      -15     
  Partials      1132     1132              
Files with missing lines Coverage Δ
src/prompts/bugbot.ts 100.00% <100.00%> (ø)
src/tooling/bugbot_benchmark_runner.ts 77.77% <100.00%> (ø)

... and 43 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@efraespada
efraespada merged commit a3f3e3e into develop Oct 28, 2026
13 checks passed
@github-project-automation github-project-automation Bot moved this from In Progress to Done in vypdev Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size: XL Indicates that the issue or task is extra large in scope and complexity. state:verified

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

🐛🧑‍💻 - Bugbot recommends an undeclared setup dart input despite the versioned upstream contract

3 participants