Skip to content

fix: ground validation findings in exact counterexamples - #762

Merged
efraespada merged 2 commits into
developfrom
bugfix/749-bugbot-claims-an-already-rejected-empty-email-domain-label-is-accepted-exact-unicode-counterexample
Oct 28, 2026
Merged

efraespada merged 2 commits into
developfrom
bugfix/749-bugbot-claims-an-already-rejected-empty-email-domain-label-is-accepted-exact-unicode-counterexample

Conversation

@efraespada

@efraespada efraespada commented Oct 28, 2026 •

Copy link
Copy Markdown
Member

Bugbot could claim that a concrete Unicode input passes validation even though an empty domain label fails the visible universal predicate. The shared review instruction now traces exact code points and separators through transformations, early returns, collection predicates and the caller to the eventual effect.

Closes #749.

What changed

  • Require proof for the exact acceptance claim while retaining supported inputs and demonstrated admitted-invalid-value defects. Ordinary/partitioned review and the benchmark share the guidance.
  • Add six fixed corpus cases and 19 distinct regressions: exact Kotlin witness/source trace, complete caller guard, normalization before checking, existential/ignored-result/late-normalization defects, five statically compiled TypeScript executors, source/emission conformance, structural guards after normalization and inert-oracle isolation. The 71 earlier cases and expected values remain unchanged.
  • Before a static fixture function runs, require exact corpus identity/text and comparison-only transpilation matching that actual function. Unicode witness values are read verbatim from checked data. The emitted corpus string is never evaluated or imported; fake-save effects come from the bound compiled function.
  • Compact redundant prose while preserving prior grounding contracts and the original 15,000-character context bound. Update SDD, catalog, public docs, evidence and bundles.

Validation

  • The three instruction-delivery regressions fail before the change; all 112 directed tests and 79 deterministic evaluation checks pass afterwards.
  • Types/lint, specs/catalog, documentation/workflows/agent docs, Graphify, build/package validation (87 files) and CLI + typed API + isolated local web smoke pass.
  • Complete global coverage: 773 suites, 14,184 tests passed, 35 existing omissions; all global/module budgets and setup acceptance passed. Local diff audit: all 2 changed executable lines covered, zero misses or uncovered modified branches. Extension: 267 tests, 100% statements/branches/functions/lines. Normal commit hooks and remote exact-head gates remain required.

Review notes

The consumer's compiled Kotlin/Java/HTTP/PostgreSQL receipts are supplied external evidence; this change performs a source-level witness trace and authored local TypeScript controls. Reference predictions test the deterministic scorer and do not measure model accuracy. No actual external-model benchmark was performed for this change. The correction is probabilistic guidance, without a finding filter, stricter character policy, permission change or future-accuracy guarantee.

Final reviewed candidate: 1ef109672c915399c7f8093f26270b4fd3b8ce0f (parent 4e2d40523a1f718a76f03d4bb9754671ec7a16d7). Normal build/full-Jest/lint hooks passed with 773 suites / 14,184 tests and 35 existing omissions. Source/bundle hashes match the final full coverage run; product code/bundles remain byte-identical to the first candidate. The updated review must verify the static-function correction and close the original finding thread before merge.

…-is-accepted-exact-unicode-counterexample: ground validation findings in exact counterexamples
@docs-page

docs-page Bot commented Oct 28, 2026

Copy link
Copy Markdown

To preview the documentation for this pull request, visit the following URL:

docs.page/vypdev/copilot~762

Documentation is deployed and generated using docs.page

@vypbot vypbot added size: XL Indicates that the issue or task is extra large in scope and complexity. state:ai-processing labels Oct 28, 2026
@vypbot
vypbot self-requested a review October 28, 2026 12:55
@vypbot vypbot added this to vypdev Oct 28, 2026
@vypbot vypbot moved this to In Progress in vypdev Oct 28, 2026
@vypbot
vypbot changed the base branch from develop to master October 28, 2026 12:55
@vypbot vypbot added the 75% Progress: 75% label Oct 28, 2026
@vypbot
vypbot changed the base branch from master to develop October 28, 2026 12:55

@vypbot vypbot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Current status: All findings originating in this review are resolved.
Last reconciled on 1ef1096. See aggregate Bugbot status.

🤖 Bugbot review snapshot

Bugbot reported 1 potential problem when commit 4e2d405 was analyzed. This snapshot is historical; use the status block above for current state. 1 finding is linked to changed code.

Findings

  • medium: Validation controls do not execute the authored fixture — src/tooling/__tests__/bugbot_benchmark.test.ts:228

@vypbot

vypbot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Bugbot: review complete

Current status: No active findings on 1ef1096.

Pull request · Verified commit · Workflow run

@vypbot

vypbot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Copilot approval · Human review needed

Status: A protected workflow, policy, or trust-boundary file changed.
Completed: 1/1 observed checks succeeded; Bugbot complete.
Action required: Ask a human reviewer to decide.
Impact: No new approval is confirmed for this revision.
See checks

Technical evidence

Revision: 1ef109672c91; base: e3917cd260f1; reason: protected-path.

@codecov-commenter

codecov-commenter commented Oct 28, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.80%. Comparing base (5b3664f) to head (1ef1096).
⚠️ Report is 43 commits behind head on develop.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop     #762      +/-   ##
===========================================
+ Coverage    94.50%   94.80%   +0.30%     
===========================================
  Files         1035     1078      +43     
  Lines        33299    34864    +1565     
  Branches      9824    10357     +533     
===========================================
+ Hits         31468    33053    +1585     
+ Misses         699      682      -17     
+ Partials      1132     1129       -3     
Files with missing lines Coverage Δ
src/prompts/bugbot.ts 100.00% <100.00%> (ø)

... and 52 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…-is-accepted-exact-unicode-counterexample: bind review controls to their compiled fixture source
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

75% Progress: 75% size: XL Indicates that the issue or task is extra large in scope and complexity. state:verified

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

🐛🧑‍💻 - Bugbot claims an already rejected empty email domain label is accepted exact Unicode counterexample

3 participants