Repository navigation
Reviewer scope gap: diff review is static — no runtime verification lane #1361
Description
The gap (found in fleet review, 2026-10-27)
The PR diff reviewers (pr-diff-review + pr-diff-review-b) are static: they read commits and diffs, verify findings against current main by reading, and file issues. They never execute the changed behavior. A regression that only appears at runtime — a config interaction, a timing/ordering effect, a behavior that typechecks and reads correctly but acts wrong — is invisible to them and is caught later, by CI if the suite happens to cover it, or by the fleet fixing it after it lands.
The fleet already pays for this gap in the other direction: several of the queue's small_fix tasks in October were follow-up fixes for behavior that a diff read as fine.
Scope (assess per repo, then build the bounded version or record why not)
- Design a bounded runtime-verification step for high-risk findings only: when a reviewer files a finding it labels high-risk (a named risk class in the packet — e.g. behavior-change-no-test, config/routing touch, data-path change), a verification pass runs the touched package's targeted tests on the laptop against the PR head (or against main + the diff's scenario where a PR head isn't runnable) and attaches the result to the finding's issue — turning "reads wrong" into "reproduces" or "does not reproduce" before a solver lane picks it up.
- Bound it hard: targeted tests only, laptop or CI only (never the sandbox), per-run cap in line with the reviewer batch caps, and no verification step for findings that are already test-covered or purely structural.
- Per-repo verdict required: for each of the four repos, either the lane operates there or the issue records why CI coverage makes it unnecessary there (suite strength, runtime surface, cost).
- Guardrail interaction: reviewers still never fix and never merge; verification output is evidence on the issue, and the queue keeps owning the fix.
Acceptance criteria
- A per-repo assessment exists (comment on this issue): CI coverage strength vs the runtime-regression classes the static review has missed, with 2–3 concrete past examples.
- The high-risk finding class is defined and the reviewer instructions name it (labels/packet field).
- A verification pass is demonstrated on at least one real finding end-to-end (laptop, targeted tests, result attached to the issue), or the per-repo record shows why it is unnecessary in every repo.
Filed by Muse (operator) 2026-10-27, from the cadence/scope review with Drew.
Metadata
Metadata
Assignees
Labels
Infrastructure, deployment, and operationsMedium: expected under 1 day of workNew feature or capability requestIntake review done: acceptance criteria, links, and normalized labels addedOrdinary priority (the default)Filed by the new-issue-creator jobTriaged and routed (see agent note)Queue packet kind: verify, reproduce, and route this item
Activity
developing-today commentedon Oct 27, 2026
CODE-461
dezren39 commentedon Nov 2, 2026
🤖 Agent note · triaged — decomposed; criterion-1 slice enqueued, criteria 2+3 routed to the assessment worker
repo-health:v1:triage:developing-today/code:issue:4605). The assessment must name the exact reviewer-instruction file(s) so the next slice has a concrete target, and must file the criteria-2+3 packet(s) (high-risk class + instruction update + demo pass) or record a no-action rationale here.14 remaining items
dezren39 commentedon Nov 3, 2026
agent-note:v1
state: triaged (coverage pass)
Coverage pass 2026-11-03: reviewed in the fleet-wide per-issue pass (drift / overlap / already-handled / stale / blockers / ordering). Verdict: still live; queued in repo-health-code-pilot as triage at priority 40 (priority/normal). If this is already handled at current main, the triage closes it with evidence; if it needs splitting, the triage files the split.
Found via: coverage pass 2026-11-03
dezren39 commentedon Nov 3, 2026
agent-note:v1
state: triaged
task 7704 (triage, lane code-j, direct) — verdict: NOT closable yet; exactly one landing step remains, recorded here with ordering.
Criterion-by-criterion, verified fresh 2026-11-03:
runtime-gap/high-risk(verification_class) and put the identical instruction edits for both lanes into dezren39/hatch-personal PR Bump checkmarx/ast-github-action from 2.0.11 to 2.0.19 #14 — still OPEN today (base main, head queue/6545-1361-high-risk-runtime-verification @55c5a3d5d). Verified against the source of truth: the live definition filescrons/hourly/pr-diff-review__interval@2h.mdandpr-diff-review-b__interval@2h.mdon hatch-personal main contain ZEROruntime-gapmatches. Until PR Bump checkmarx/ast-github-action from 2.0.11 to 2.0.19 #14 merges AND the two live jobs are synced from the merged definitions (cron definitions are truth: definition first, then cron.update the live pr-diff-review / pr-diff-review-b jobs, then commit), no reviewer run can emit the class — the lane does not exist operationally.bun run test -- src/collab.test.ts(pkgs/id/web, 6 tests pass) at PR Fix #1368: cap collab retries until Init #3734 head b1b28d1 on the laptop, reproduce/does-not-reproduce evidence attached to id(web): collab reconnect cap defeated by onopen reset — accept-then-drop server loops forever, never surfaces error #1368 and here; reviewer guardrails held (no fix, no merge, PR untouched).Blocker / ordering: (i) merge hatch-personal PR #14 (owner/Drew-side — it rewrites two live cron job bodies, so it lands as a config change, not as part of this code-queue triage); (ii) cron.update the live pr-diff-review and pr-diff-review-b jobs from the merged definitions; (iii) then close this issue citing criterion 2 landed + the criterion 1/3 evidence above. No code-repo work remains for #1361 itself, and no new fix packet is enqueued from this triage — the remaining step lives in hatch-personal, outside this queue's repo scope. This triage task completes as triaged with the blocker recorded; the issue stays open as the tracker until step (iii).