Repository navigation
Meta-reviewer: grade the reviewers, score model/effort choices, and evolve the reviewer guidelines #1340
Description
Work (Drew, 2026-10-27)
A daily meta-reviewer job (pool-meta-reviewer) reviews the
reviewers — the quota steward, the lane shifts, the dispatcher, the
babysitter, the diff reviewers — and closes the learning loop:
- Audit the steward's change ledger: every change has an event,
every event has a measured outcome, predictions vs actuals scored.
Changes that didn't do what they claimed get named, with the
evidence. - Grade model/effort choices by outcome across the window: per pool
and per effort level — completion rate, stall rate, rework
(reopened issues, follow-up fixes), cost per merged PR, wall-clock
per task. Answer, with data, the standing questions: when does
model-up + effort-down beat model-down + effort-up? Where is max
effort buying nothing (equal outcomes at high)? Where is a pool
failing tasks a stronger pool would have finished? - Update the reviewer guidelines (living doc in the goal
workspacefiles/) from the evidence: routing defaults, effort
assignment heuristics, nudge/steward thresholds. Guideline changes
are dated, evidence-linked, and themselves reviewed next cycle —
the guidelines carry a changelog the meta-reviewer appends to. - Escalate to Drew only decisions that are genuinely his (new pool,
new provider, spend-class change); everything inside the POOL MAP
is the fleet's to tune and report.
Acceptance criteria
- Job live; first daily review produced with per-pool/per-effort
outcome tables from real fleet data. - Guidelines doc exists with its first evidence-based revision
and changelog. - At least one standing question (max-effort waste, model-up/
effort-down crossover) answered with fleet numbers, or an
explicit "insufficient data, collecting via ".
Child of the pool & routing intelligence epic. Filed by Muse
(operator) 2026-10-27.
Metadata
Metadata
Assignees
Labels
Infrastructure, deployment, and operationsMedium: expected under 1 day of workNew feature or capability requestIntake review done: acceptance criteria, links, and normalized labels addedOrdinary priority (the default)Filed by the new-issue-creator jobTriaged and routed (see agent note)Queue packet kind: verify, reproduce, and route this item
Activity
developing-today commentedon Oct 27, 2026
CODE-444
3 remaining items
dezren39 commentedon Nov 3, 2026
agent-note:v1
state: triaged (coverage pass)
Coverage pass 2026-11-03: reviewed in the fleet-wide per-issue pass (drift / overlap / already-handled / stale / blockers / ordering). Verdict: still live; queued in repo-health-code-pilot as triage at priority 40 (priority/normal). If this is already handled at current main, the triage closes it with evidence; if it needs splitting, the triage files the split.
Found via: coverage pass 2026-11-03
dezren39 commentedon Nov 3, 2026
agent-note:v1
state: picked_up
Picked up queue task 7682 (triage #1340) on the OpenChamber scheduled lane (lane 7b47536b-c162-444c-be8a-3a9538cc4cb5, worker repo-worker-code).
This is a re-triage, not a restart: the 2026-10-28 lane-A triage (task 1324) already declared the design live and routed the single remainder to code#1582. I am revalidating that verdict against current state rather than repeating the decomposition.
Found via: queue claim (coverage pass 2026-11-03)
dezren39 commentedon Nov 3, 2026
🤖 Agent note · triaged (re-triage; verdict: design live and iterating, one real criterion still unmet and already routed)
repo-worker-code, OpenChamber scheduled lane7b47536b-c162-444c-be8a-3a9538cc4cb5, run 2026-11-03 ~15:00ZAcceptance criteria, verified 2026-11-03T15:10Z
pool-meta-reviewerhas completed eight daily runs; latest ishidden_files/meta-review/2026-11-03-report.md(+-evidence.md) for the window 2026-11-02T13:12Z–2026-11-03T13:12Z. It is doing the charged work: steward ledger audit with named corrections, prediction scoring, per-pool walls/rotation analysis.files/reviewer-guidelines.md(49 KB, last revised 2026-11-03T07:56 local) carries an append-only changelog; the 2026-11-03 run appended four dated, evidence-linked entries (telemetry graded by adoption share; disabled decision-maker graded on active span; expired closures; nudges graded on outcomes).Where the remainder actually stands (this is what changed since 2026-10-28)
The prior triage routed the telemetry remainder to #1582, expecting collection to start on landing. It landed — #1582 CLOSED COMPLETED 2026-11-02T22:18:19Z, and I verified the mechanism locally:
repo-task/v1now defines$defs.completion_telemetry(pool,model,effort,cost_usd, all optional/additive), andrepo-queue.pycarriesTELEMETRY_FIELDSplus a--telemetryargument oncomplete.But collection did not start: 846 tasks completed after the landing and exactly 1 carries any telemetry key, null-valued. That gap is now filed as #4919 (created 2026-11-03T13:47Z, OPEN, already
queue/queued— its own task exists). So the blocker for criterion 1 has moved from "mechanism missing" (#1582) to "adoption missing" (#4919). The 2026-10-28 prediction is superseded, not wrong: the mechanism exists, workers don't pass the fields.Blocker and ordering
complete(its queue task).pool/model/effort.No implementation work remains on #1340 itself. It stays open as the parent tracker for that one dependent step; closing it now would strand criterion 1 with no tracker. No Drew escalation: everything is inside the POOL MAP the fleet may tune, per the issue's own escalation rule.
~/hatch-personal/workspace/goals/keep-the-four-repos-healthy/on this host; the~/workspace/goals/...copy is trimmed and has nofiles/orhidden_files/meta-review/, so do not read the absence there as missing artifacts.files/reviewer-guidelines.md· reporthidden_files/meta-review/2026-11-03-report.md