Deterministic self-healing drift: remove the LLM from remediation

CLAUDE.md

@@ -36,10 +36,13 @@ entire repo, not just staged files.
 
 ## Drift Remediation
 
-Automated drift remediation lives in `scripts/`:
+There is no LLM/agent in the remediation loop. Automated remediation is
+deterministic and scoped to model-family churn only; general drift is caught by
+the daily drift test and fixed by a human. Lives in `scripts/`:
 
 - `scripts/drift-report-collector.ts` — runs drift tests, produces `drift-report.json`
-- `scripts/fix-drift.ts` — reads drift report, invokes Claude Code to fix builders, creates PR or issue
+- `scripts/drift-sync.ts` — zero-LLM model-family sync (mechanical registry edit or needs-human note file) + reusable git/PR plumbing
+- `scripts/drift-sync-check.ts` — the deterministic gate (changed-file allowlist, checksum-pin re-assert, clean re-collect) replacing remediation-diff review
 
 See `DRIFT.md` for full documentation and `.github/workflows/fix-drift.yml` for the CI workflow.
 

DRIFT.md

@@ -180,26 +180,33 @@ See `.github/workflows/test-drift.yml`.
 
 ## Automated Drift Remediation
 
-When the daily drift test detects critical diffs on the `main` branch, the `fix-drift.yml` workflow runs automatically:
-
-1. **Collect** — `scripts/drift-report-collector.ts` runs drift tests and produces a structured `drift-report.json`
-2. **Fix** — `scripts/fix-drift.ts` (default mode) constructs a prompt from the report and invokes Claude Code to fix the builders
-3. **Verify** — Independent `pnpm test` and `pnpm test:drift` steps confirm the fix works
-4. **PR** — `scripts/fix-drift.ts --create-pr` stages and commits the changes, bumps the version, and opens a pull request
-5. **Issue** (on failure) — `scripts/fix-drift.ts --create-issue` opens a GitHub issue with the drift report and Claude Code output
-
-Steps 2 and 4/5 are separate invocations of `fix-drift.ts` with different modes.
+There is no LLM/agent in the remediation loop. General (non-model-churn) drift
+is **not** auto-fixed by anything — it is caught by the daily drift test (which
+alerts on its own; see above) and fixed by a human like any other bug. The only
+automated remediation is the deterministic, zero-LLM **model-family sync**,
+which handles exactly one class of drift: a provider adding or retiring a
+model family. The `fix-drift.yml` workflow runs it on `workflow_dispatch`, a
+daily **scheduled cron** (independent of drift-test failure — a retired model
+family does not, by itself, fail the drift tests), and on a failed `Drift
+Tests` run (an opportunistic attempt in case the failure was model churn):
+
+1. **Sync** — `scripts/drift-sync.ts` fetches each provider's live `/models` listing directly and diffs it against the frozen classification in `src/__tests__/drift/model-registry.ts`:
+   - a classified family absent from live listings with **zero remaining aimock references** → a mechanical, comment-marked removal
+   - a still-referenced deprecated family, or a genuinely new/unclassified family → **never** auto-edited; a family-keyed dedup note file is written under `drift-proposals/` and the run is routed to a human (no PR spam on re-fire)
+2. **Gate** — `scripts/drift-sync-check.ts` re-verifies any mechanical edit before (inside `drift-sync.ts`) and after (workflow defense-in-depth) it is kept: a changed-file allowlist (only `model-registry.ts` data literals + `drift-proposals/` notes), a checksum-pin re-assert over the frozen classification logic, and a clean re-collect
+3. **PR** — the workflow always opens a pull request that a human reviews + merges (no auto-merge). There are two distinct PR classes:
+   - **`ok-applied`** — a successful mechanical registry edit. Pushed onto the `fix/drift-*` branch `drift-sync.ts` committed onto; a human reviews CI + the diff and merges.
+   - **`needs-human`** — a routed decision. `drift-sync.ts` commits the `drift-proposals/` note file(s), and the workflow pushes a **distinct `drift-needs-human/*` branch** and opens a PR so the note lands in the repo (the job also goes RED + Slack-alerts so the decision is seen). The PR is **never auto-merged**. To approve a _new-family_ note, set its `Decision: include` line and **merge the PR**; the **next** drift-sync run reads the approved note from `main` and applies the mechanical registry edit (an `ok-applied` PR). That two-run hand-off is how the loop closes.
+
+   **Re-fires never spam a second PR — idempotent in every run shape.** Because a drift-sync PR is never auto-merged, an un-merged drift is re-detected on every daily cron run. Both PR classes therefore dedup on a **stable changeset key**: `drift-sync.ts` emits a date-independent `changeset-key` (a hash of the sorted set of applied + deferred family outcomes, independent of the date-stamped comment text and the run-id branch name), and each PR body carries a `<!-- drift-changeset: <key> -->` marker. Before opening a PR, the workflow skips if an open PR already carries that marker. This covers the **mixed run** — a mechanical removal of one family committed the same run a _different_ family is deferred to a human (its note already on `main`) — whose committed diff is a registry edit with **no new note file**: a note-path-only key would be empty there and let a new PR open every day. A run that produces no new commit at all (note already on `main`, nothing applied) pushes nothing; and the older per-note `drift-proposal-note: <path>` body marker is retained as a secondary guard.
 
 ### Artifacts
 
-Both workflows upload artifacts:
-
-- `drift-report.json` — structured drift data (retained 30 days)
-- `claude-code-output.log` — Claude Code's reasoning and tool calls (fix workflow only)
+- `drift-report.json` (test-drift.yml) / `drift-sync-log`, `drift-sync-check-log` (fix-drift.yml) — structured/plaintext run output (retained 30 days)
 
 ### Manual trigger
 
-The fix workflow also supports `workflow_dispatch` for manual runs.
+The sync workflow also supports `workflow_dispatch` for manual runs.
 
 ## Cost
 

scripts/drift-report-collector.ts

@@ -633,6 +633,28 @@ export function extractRawLocation(msg: string): string {
   );
 }
 
+/**
+ * F1 — PER-LEG (single-message) infra classification. This is the primitive
+ * that keeps ONE leg's failure from batch-poisoning its siblings: each
+ * unparseable failure is judged ALONE, so an unparseable leg can never flip a
+ * genuine infra leg into the quarantine lane (nor a benign infra leg mask a
+ * genuinely-unparseable sibling).
+ *
+ * A message is benign infra iff it carries POSITIVE infra evidence AND shows no
+ * drift-like signal, both judged on the SAME stack-stripped text (A3 — the two
+ * scans must never disagree because they saw different inputs). A bare
+ * AssertionError with no infra token is NOT infra (returns false → the caller
+ * quarantines it for review), preserving the CLASS 1 "unrecognized ⇒ never a
+ * false all-clear" invariant at the per-leg grain.
+ */
+export function classifySingleUnparseableAsInfra(message: string): boolean {
+  // Normalize ONCE; both scans consume the identical normalized text (A3).
+  const normalized = stripStackFrames(message);
+  const hasInfraEvidence = INFRA_INDICATORS.some((re) => re.test(normalized));
+  const isDriftLike = DRIFT_LIKE_INDICATORS.some((re) => re.test(normalized));
+  return hasInfraEvidence && !isDriftLike;
+}
+
 export function classifyUnparseableAsInfra(unparseableMessages: string[]): boolean {
   // CLASS 1 — fail-loud on absent evidence. No messages means NO positive infra
   // evidence, so this is NOT a benign "all clear". `[].every(...)` is vacuously
@@ -641,17 +663,13 @@ export function classifyUnparseableAsInfra(unparseableMessages: string[]): boole
   // false all-clear.
   if (unparseableMessages.length === 0) return false;
 
-  // Normalize ONCE per message; both scans consume the identical normalized
-  // text (A3 — the two scans must never disagree due to differing inputs).
-  const normalized = unparseableMessages.map(stripStackFrames);
-
-  // Infra requires POSITIVE evidence on EVERY message AND zero drift signal on
-  // ALL of them. If any message lacks an infra indicator, or any message looks
-  // drift-like, we do NOT swallow — the caller throws (exit 1, investigate).
-  const allInfraErrors = normalized.every((msg) => INFRA_INDICATORS.some((re) => re.test(msg)));
-  const anyDriftLike = normalized.some((msg) => DRIFT_LIKE_INDICATORS.some((re) => re.test(msg)));
-
-  return allInfraErrors && !anyDriftLike;
+  // Batch semantics are exactly the conjunction of the per-leg predicate: the
+  // whole set is benign infra iff EVERY message is individually benign infra
+  // (every message has an infra indicator AND none is drift-like). Kept for the
+  // back-compat batch callers/tests; the collector's swallow-vs-quarantine
+  // decision below is now made PER LEG via classifySingleUnparseableAsInfra so a
+  // mixed batch no longer drags benign infra legs into quarantine.
+  return unparseableMessages.every(classifySingleUnparseableAsInfra);
 }
 
 /**
@@ -927,21 +945,33 @@ export function collectDriftEntries(results: VitestJsonResult): CollectResult {
       console.warn(`  Unparseable failure message (first 300 chars): ${msg.slice(0, 300)}`);
     }
 
-    if (classifyUnparseableAsInfra(unparseableMessages)) {
+    // F1: classify EACH unparseable failure on its own so one leg's genuinely-
+    // unparseable output never batch-poisons a sibling that failed on benign
+    // infra. Previously a single all-or-nothing classifyUnparseableAsInfra call
+    // over the whole batch meant one non-infra leg flipped the ENTIRE batch to
+    // "not infra" and dragged every benign infra sibling into quarantine (exit
+    // 5) — poisoning the shared base report for every downstream PR. Now an infra
+    // leg is swallowed on its own merits and only the genuinely-unparseable
+    // leg(s) are quarantined for review. A1.3: quarantine (exit 5), never a
+    // fail-loud crash (exit 1) — exit 1 is reserved for genuine collector bugs.
+    const infraCount = unparseableFailures.filter((f) =>
+      classifySingleUnparseableAsInfra(f.message),
+    ).length;
+    const quarantinedLegs = unparseableFailures.filter(
+      (f) => !classifySingleUnparseableAsInfra(f.message),
+    );
+    if (infraCount > 0) {
       console.warn(
-        `WARNING: ${unparseable} test failure(s) appear to be API/infrastructure errors ` +
-          `(not drift reports). Continuing with 0 drift entries.`,
+        `WARNING: ${infraCount} test failure(s) appear to be API/infrastructure errors ` +
+          `(not drift reports) — swallowed as benign infra.`,
       );
-    } else {
-      // A1.3: genuine-but-unparseable drift is no longer a fail-loud crash (exit
-      // 1). Each such failure is quarantined (exit 5) so it surfaces for human
-      // review without being silently swallowed as a green. Exit 1 is now
-      // reserved for genuine collector bugs (unhandled exceptions).
+    }
+    if (quarantinedLegs.length > 0) {
       console.warn(
-        `WARNING: ${unparseable} test failure(s) could not be parsed as drift reports — ` +
+        `WARNING: ${quarantinedLegs.length} test failure(s) could not be parsed as drift reports — ` +
           `quarantined for review (exit 5).`,
       );
-      for (const f of unparseableFailures) {
+      for (const f of quarantinedLegs) {
         quarantine.push({
           provider: "unknown",
           testName: f.testName,

scripts/drift-slack-summary.ts

@@ -26,7 +26,7 @@ import { appendFileSync, existsSync } from "node:fs";
 import { resolve } from "node:path";
 import { fileURLToPath } from "node:url";
 
-import { readDriftReport } from "./fix-drift.js";
+import { readDriftReport } from "./drift-sync.js";
 import type { DriftEntry, DriftReport, DriftSeverity } from "./drift-types.js";
 
 // ---------------------------------------------------------------------------