Drift subsystem cleanup: prune inert code, pin excludeFamilies, mirror-equivalence guard, trim workflow perms#336
Merged
Merged
Conversation
commit: |
jpr5
added a commit
that referenced
this pull request
Jul 24, 2026
## Needs a human decision (drift-sync) The deterministic, zero-LLM drift-sync found a model-family change it must NOT auto-apply (a genuinely new/unclassified family, a still-referenced deprecation, or a registry structural mismatch). It wrote the note file(s) below and opened this PR so the decision is REACHABLE in the repo. > **Not auto-merged — a human decides.** > >⚠️ **The `Decision: include` instructions in this template do NOT apply to this > PR.** That protocol is scoped to a *new-family* note only: `renderProposalNote` > emits the `## Decision` block solely under `kind === "new-family"`, and > `parseProposalDecision` has exactly one call site — the addition half of > `runDriftSyncCore`. The deprecation half never reads a decision at all. > > Both notes in this PR are **still-referenced deprecations**, and they contain > **no `Decision:` line whatsoever**. Writing one would be a no-op. There is no > marker, of any value, that makes a later run mechanically resolve this note > class — only a human registry edit can. **That edit is now part of this PR** > (see *Human decision* at the bottom), so there is no two-run hand-off here: > merging this PR both records the decision and resolves the drift. > > (For reference, the original template text: for a *new-family* note you would > set `Decision: include` to classify it, or delete the note / close the PR to > reject, then merge — and the next run would apply the registry edit itself.) ### Note file(s) in this PR <!-- drift-changeset: e08d8fcc953752d5 --> - `drift-proposals/gemini-gemini-1.5-flash-deprecated-referenced.md` <!-- drift-proposal-note: drift-proposals/gemini-gemini-1.5-flash-deprecated-referenced.md --> - `drift-proposals/gemini-gemini-1.5-pro-deprecated-referenced.md` <!-- drift-proposal-note: drift-proposals/gemini-gemini-1.5-pro-deprecated-referenced.md --> ### drift-sync outcome ``` npm warn Unknown project config "minimum-release-age". This will stop working in the next major version of npm. See `npm help npmrc` for supported config options. npm warn Unknown project config "block-exotic-subdeps". This will stop working in the next major version of npm. See `npm help npmrc` for supported config options. needs-human note(s) written this run — routed to human without a live re-collect (no registry edit to re-verify) [needs-human-still-referenced] gemini/gemini-1.5-flash: "gemini-1.5-flash" is deprecated but still referenced in source — routed to human (drift-proposals/gemini-gemini-1.5-flash-deprecated-referenced.md) [needs-human-still-referenced] gemini/gemini-1.5-pro: "gemini-1.5-pro" is deprecated but still referenced in source — routed to human (drift-proposals/gemini-gemini-1.5-pro-deprecated-referenced.md) [skipped] anthropic: live /models listing too short to trust for anthropic (10 raw id(s), need >= 19 — the number of families aimock mocks for this provider) — never mass-removing off a truncated or empty listing npm warn Unknown env config "block-exotic-subdeps". This will stop working in the next major version of npm. See `npm help npmrc` for supported config options. npm warn Unknown env config "minimum-release-age". This will stop working in the next major version of npm. See `npm help npmrc` for supported config options. npm warn Unknown project config "minimum-release-age". This will stop working in the next major version of npm. See `npm help npmrc` for supported config options. npm warn Unknown project config "block-exotic-subdeps". This will stop working in the next major version of npm. See `npm help npmrc` for supported config options. ^[[33m[STARTED]^[[39m Backing up original state... ^[[32m[COMPLETED]^[[39m Backed up original state in git stash (c62c488) ^[[33m[STARTED]^[[39m Running tasks for staged files... ^[[33m[STARTED]^[[39m package.json^[[0;90m — 2 files^[[0m ^[[33m[STARTED]^[[39m *.{ts,mts,js,mjs,cjs,json,html,css,md}^[[0;90m — 2 files^[[0m ^[[33m[STARTED]^[[39m *.{ts,mts,js,mjs}^[[0;90m — 0 files^[[0m ^[[33m[SKIPPED]^[[39m *.{ts,mts,js,mjs}^[[0;90m — no files^[[0m ^[[33m[STARTED]^[[39m prettier --write ^[[32m[COMPLETED]^[[39m prettier --write ^[[32m[COMPLETED]^[[39m *.{ts,mts,js,mjs,cjs,json,html,css,md}^[[0;90m — 2 files^[[0m ^[[32m[COMPLETED]^[[39m package.json^[[0;90m — 2 files^[[0m ^[[32m[COMPLETED]^[[39m Running tasks for staged files... ^[[33m[STARTED]^[[39m Applying modifications from tasks... ^[[32m[COMPLETED]^[[39m Applying modifications from tasks... ^[[33m[STARTED]^[[39m Cleaning up temporary files... ^[[32m[COMPLETED]^[[39m Cleaning up temporary files... [fix/drift-2026-07-24-30074183288 85fc683] fix(drift-sync): mechanical model-family sync (needs-human note file(s)) 2 files changed, 14 insertions(+) create mode 100644 drift-proposals/gemini-gemini-1.5-flash-deprecated-referenced.md create mode 100644 drift-proposals/gemini-gemini-1.5-pro-deprecated-referenced.md reason=needs-human changeset-key=e08d8fcc953752d5 ``` --- ## Human decision: keep mocking these families **Decision: KEEP.** `gemini-1.5-pro` and `gemini-1.5-flash` are gone from Gemini's live `/models` listing, but aimock still mocks them on purpose — clients pinned to older SDKs and older CopilotKit configs still send those ids, and `isReasoningModel()` must keep answering `false` for them. So nothing is being removed. Instead, commit `a3dc250` records the decision in the registry by moving both families from `includeFamilies.gemini` into `excludeFamilies.gemini`'s existing **"Retired / legacy specialty"** bucket, and re-pins the two affected `DATA_FROZEN` membership hashes. `excludeFamilies` means *"not counted as text-generation **drift**"*, not *"not mocked"* — `gemini-pro`, `gpt-3.5` and `gemini-2.0-flash-exp` already live there while being mocked and heavily referenced in `src/`. Prior art for exactly this move: #315 (`fix(drift): classify bare chat-latest alias in openai excludeFamilies`). Because `detectDeprecatedFamiliesForSync` draws its candidate set from `includeFamilies[provider]`, the two families can no longer become deprecation candidates — drift-sync stops re-flagging them needs-human, which ends the daily red job and the daily Slack page. The two note files stay in this PR as the audit trail for the `excludeFamilies` entries; they become inert on merge and will never be re-read or re-written. **Nothing that references `gemini-1.5-*` was touched.** `src/model-utils.ts:54` (`NONREASONING_FAMILIES`), the three gemini reasoning-capability suites, the `gemini.test.ts` / `vertex-ai.test.ts` conformance assertions and the recorded `sdk-shapes.ts` fixture are all unchanged. ### Diff ``` src/__tests__/drift/logic-pin.test.ts | 4 ++-- src/__tests__/drift/model-registry.ts | 12 +++++++++--- 2 files changed, 11 insertions(+), 5 deletions(-) ``` ### Known side effect Gemini's truncated-listing trust floor is `includeFamilies[provider].size`, so it drops from 11 to 9. Reversible: move the families back and re-pin. --- ## Local red-green proof ### 1. Baseline — before any edit, drift suites all green ``` $ pnpm exec vitest run src/__tests__/drift/ ✓ src/__tests__/drift/model-registry.ts ... (all 9 files) Test Files 9 passed (9) Tests 87 passed (87) ``` ### 2. RED — registry membership moved, pins NOT yet updated `logic-pin.test.ts` fails loudly on the stale membership pins, exactly as its module doc promises: ``` $ pnpm exec vitest run src/__tests__/drift/logic-pin.test.ts × classification-data membership freeze > freezes includeFamilies.gemini membership → Frozen data set "includeFamilies.gemini" membership changed (now: ["gemini-2.0-flash","gemini-2.0-flash-lite","gemini-2.5-flash", "gemini-2.5-flash-lite","gemini-2.5-pro","gemini-3.1-flash-lite", "gemini-3.5-flash","gemini-3.5-flash-lite","gemini-3.6-flash"]). Expected: "4a9428b64ffcff0fbb79878d88ed993ffac53e43d64e26bcf5d86509626f593d" Received: "c2e2c56b8f8d5fc56152b4633e7d3782e95b7eeb9bc123da71f00e884a54a743" × classification-data membership freeze > freezes excludeFamilies.gemini membership → Frozen data set "excludeFamilies.gemini" membership changed (now: ["aqa","gemini-1.5-flash","gemini-1.5-pro","gemini-2.0-flash-exp", ...]). Expected: "e3545138234ad782937f66760bef942fca7d4bd0934a87da30bf6e5816ba69b1" Received: "c95dedab7588212bbba0bf9ab6434bfd43a449adbe7b35f81f48271ee849a9c2" Test Files 1 failed (1) Tests 2 failed | 18 passed (20) ``` The two new pin values were derived independently — `sha256(JSON.stringify([...set].sort()))` recomputed from `model-registry.ts` in a standalone script, matching the values the test itself reports: ``` includeFamilies.gemini c2e2c56b8f8d5fc56152b4633e7d3782e95b7eeb9bc123da71f00e884a54a743 excludeFamilies.gemini c95dedab7588212bbba0bf9ab6434bfd43a449adbe7b35f81f48271ee849a9c2 ``` ### 3. GREEN — pins re-pinned, same command ``` $ pnpm exec vitest run src/__tests__/drift/logic-pin.test.ts ✓ src/__tests__/drift/logic-pin.test.ts (20 tests) 3ms Test Files 1 passed (1) Tests 20 passed (20) ``` ### 4. Full suite — zero regressions ``` $ pnpm test Test Files 165 passed | 1 skipped (166) Tests 4724 passed | 47 skipped (4771) ``` And the suites that specifically pin `gemini-1.5-*` behavior: ``` $ pnpm exec vitest run src/__tests__/model-utils.test.ts \ src/__tests__/gemini-reasoning-capability.test.ts \ src/__tests__/gemini-audio-reasoning-capability.test.ts \ src/__tests__/gemini-toolonly-reasoning-capability.test.ts \ src/__tests__/gemini.test.ts src/__tests__/vertex-ai.test.ts \ src/__tests__/drift/ src/__tests__/drift-sync-core.test.ts Test Files 16 passed (16) Tests 236 passed (236) ``` ### 5. Drift resolution proved offline `scripts/drift-sync.ts` has no `--dry-run` / `--check` flag, so the canary was exercised directly instead: `detectDeprecatedFamiliesForSync` is exported and pure, so it was called against a simulated live `/models` listing containing every gemini family aimock classifies **except** `gemini-1.5-pro` / `gemini-1.5-flash` (31 ids — comfortably over the trust floor under both the old size 11 and the new size 9), once on the pre-fix tree and once on the fixed tree. Pre-fix (this branch at `85fc683`) — reproduces the needs-human routing: ``` includeFamilies.gemini.size (trust floor) = 11 simulated live listing length = 31 status = checked candidates = [{"provider":"gemini","family":"gemini-1.5-flash","stillReferenced":true}, {"provider":"gemini","family":"gemini-1.5-pro","stillReferenced":true}] gemini-1.5 candidates = 2 needs-human candidates = 2 >>> STILL FLAGGED (red) ``` Post-fix (`a3dc250`) — resolved: ``` includeFamilies.gemini.size (trust floor) = 9 simulated live listing length = 31 status = checked candidates = [] gemini-1.5 candidates = 0 needs-human candidates = 0 >>> RESOLVED: gemini-1.5 NOT flagged ``` `detectDeprecatedFamiliesForSync` is the exact function `drift-sync.ts` uses to build the deprecation candidate set, so this is the real routing surface, not a stand-in. **Plus a genuine LIVE confirmation from this PR's own CI.** The `drift-live-pr` check ran the drift suite against the real provider `/models` listings on this head commit (`preflight: Gemini key OK`) and the uploaded head drift report contains exactly one entry — a pre-existing *anthropic* `claude-opus-5` unclassified-family finding, also present in the base report — with **zero** `gemini-1.5` mentions and no gemini entry at all: ``` Drift gate PASSED: 1 advisory (pre-existing) finding(s) surfaced, none new-in-head. ``` That confirms against the live listing that moving both families to `excludeFamilies` introduced no new drift and no unclassified-family regression (`isClassifiedFamily` is satisfied by include **or** exclude). To be precise about what it does *not* prove: this leg does not surface the still-referenced-deprecation class as a report entry — the base report shows zero `gemini-1.5` mentions too — so the live job is regression evidence, and the needs-human routing fix itself is proved by the `detectDeprecatedFamiliesForSync` red-green above. ### 6. Pre-push gates ``` $ pnpm format:check → All matched files use Prettier code style! $ pnpm lint → clean (no output) $ pnpm exec tsc --noEmit → clean (no output) $ pnpm build → ✔ Build complete (273 files) $ pnpm test:exports → No problems found 🌟 ``` ### 7. Zero behavior change → no version bump, no changeset The test that matters is not "which directory did the files sit in" but **does this change aimock's behavior for a consumer?** It does not, and that is verified three independent ways. **(a) The registry is not reachable from shipped code.** Every importer of `src/__tests__/drift/model-registry.ts` is a test or the maintenance script: ``` scripts/drift-sync.ts (repo script, not published) src/__tests__/drift-sync-core.test.ts src/__tests__/drift-sync-mirror-equivalence.test.ts src/__tests__/drift/logic-pin.test.ts src/__tests__/drift/model-registry.test.ts src/__tests__/drift/models.drift.ts ``` No shipped-source importer exists — `src/model-utils.ts`, `src/server.ts`, the handlers and every published entrypoint in `tsdown.config.ts` are all absent from that list. `includeFamilies` / `excludeFamilies` are drift-detection *metadata*, not a runtime allowlist. Confirmed at the artifact level too: `dist/` contains zero occurrences of `includeFamilies`, `excludeFamilies` or `isClassifiedFamily`, so the data isn't inlined either, and `npm pack --dry-run` ships 576 files with **none** matching `__tests__` / `/drift/` / `model-registry` / `logic-pin` / `src/` (`files` allowlist is `["dist","fixtures",".claude-plugin","skills","CHANGELOG.md"]`; `scripts/` ships 0 entries; no `.npmignore`). **(b) The published artifact is byte-identical.** Built `dist/` from the pre-fix and post-fix trees and compared all 285 shipped code/declaration files. Note the build is **not** reproducible — two builds of *identical* source differ in declaration sourcemaps and in tsdown's non-deterministic import-alias numbering (`node_http0` ↔ `node_http1`), so a naive whole-dist hash gives a false positive. Normalizing that noise, with two identical-source builds as the control noise floor: ``` control: two IDENTICAL-source builds → files=285 differing=0 subject: PRE-fix vs POST-fix build → files=285 differing=0 VERDICT: PUBLISHED ARTIFACT SEMANTICALLY IDENTICAL ``` **(c) Consumer-visible behavior is bit-identical.** Probed the *built* `dist/` before and after with real requests for both ids — `generateContent`, SSE `streamGenerateContent`, strict-on and strict-off reasoning, the OpenAI-compatible `/v1/chat/completions` surface, and `/v1/models` — plus the `isReasoningModel()` verdicts and the captured capability warnings. The two JSON captures compare **equal**: ``` PRE isReasoningModel: {'gemini-1.5-pro': False, 'gemini-1.5-flash': False} POST isReasoningModel: {'gemini-1.5-pro': False, 'gemini-1.5-flash': False} controls (unchanged): gemini-2.5-pro=True, gemini-pro=True, some-future-model=True 8/8 request probes identical (all HTTP 200, identical bodies) capability warnings identical, still naming both gemini-1.5 ids identical overall: True ``` So aimock still mocks both ids identically, `isReasoningModel()` still returns `false` for both, and the advertised model list is unchanged. **Release mechanism.** `publish-release.yml` triggers on push to `main`, compares `package.json`'s version against npm, and publishes + tags only when that version is not yet published — i.e. releases come from a **manual `package.json` version bump**. There is no Changesets setup (no `.changeset/`, no `@changesets/cli`), no release-please and no semantic-release, and no workflow derives a version from commit messages, so **the conventional-commit type does not trigger a release**; the `fix(drift):` prefix here matches the repo's own prior art for registry classification changes (#315, #322) and is inert with respect to versioning. `publish-commit.yml` → `pkg-pr-new` is just the per-PR preview publish behind the "Continuous Releases" / "preview" checks and gates nothing on versioning. **No changeset file and no version bump are required.** Minor pre-existing caveat: `unreleased-check.yml` counts commits touching `src/` since the last tag without excluding `src/__tests__/`, so on merge it will emit a `::warning::` that the version wasn't bumped. It is a warning only — no `exit 1` — so it does not fail CI. #336 had the same shape. --- ## Separate latent defect, filed for follow-up (NOT introduced by this PR) The *mechanical* registry-edit path documented for new-family notes is currently dead-locked by its own gate, so `ok-applied` can never happen: gate-2 of `drift-sync-check.ts` re-runs `logic-pin.test.ts` as a subprocess, that file's `DATA_FROZEN` block freezes set **membership**, and a mechanical add/remove *is* a membership change — so it trips the pin, gets reverted by `revertFiles`, and reports `gate-failed`. #335 shipped the mechanical path; #336 added `DATA_FROZEN` 40 minutes later. It has never been observed because no `ok-applied` run has ever occurred. The `ok-applied` test injects a fake gate (`makeFakeDeps`), so it asserts `runSyncCheck` was *called*, never that the real pin would pass. Worth fixing separately — either exempt the `DATA_FROZEN` membership hashes from gate-2, or have drift-sync re-pin the hash as part of the allowlisted data edit and lean on human PR review as the guard.
jpr5
added a commit
that referenced
this pull request
Jul 24, 2026
… chat) (#339) ## What was unclassified `claude-opus-5` — **Claude Opus 5** — is an unclassified Anthropic model family on `main`. It appears in Anthropic's live `GET /v1/models` listing but is in neither `includeFamilies.anthropic` nor `excludeFamilies.anthropic`, so the live `/models` family canary flags it as a new-family decision and re-routes it to a human on every run. It was simply missed: the last classification wave (`34056c2`) covered the 2026-07-16 listing, and Opus 5 landed after it (`created_at: 2026-07-24`). **Only this one family is unclassified.** The live listing returns 11 ids and every other one is already classified — including the other two Claude-5 members, `claude-sonnet-5` and `claude-fable-5`. So this is a one-family fix, not a wave. ## Evidence it was unclassified Live listing (`GET https://api.anthropic.com/v1/models`, fetched 2026-07-24, key sourced from the session secret cache — never persisted anywhere): ``` count 11 has_more False claude-opus-5 | Claude Opus 5 | 2026-07-24T00:00:00Z <-- unclassified claude-sonnet-5 | Claude Sonnet 5 | 2026-06-29T00:00:00Z claude-fable-5 | Claude Fable 5 | 2026-06-07T00:00:00Z claude-opus-4-8 | Claude Opus 4.8 | 2026-05-28T00:00:00Z claude-opus-4-7 | Claude Opus 4.7 | 2026-04-14T00:00:00Z claude-sonnet-4-6 | Claude Sonnet 4.6| 2026-02-17T00:00:00Z claude-opus-4-6 | Claude Opus 4.6 | 2026-02-04T00:00:00Z claude-opus-4-5-20251101 | Claude Opus 4.5 | 2025-11-24T00:00:00Z claude-haiku-4-5-20251001 | Claude Haiku 4.5 | 2025-10-15T00:00:00Z claude-sonnet-4-5-20250929 | Claude Sonnet 4.5| 2025-09-29T00:00:00Z claude-opus-4-1-20250805 | Claude Opus 4.1 | 2025-08-05T00:00:00Z ``` Independently corroborated by PR #337's `drift-live-pr` job, whose uploaded head **and** base drift reports both contain exactly one entry — this same finding — with `conclusion: critical`. It passed #337's delta gate only because that gate is new-in-head-only, so the finding has been sitting on `main` unaddressed. ## Classification chosen, and why **`includeFamilies.anthropic`** (`src/__tests__/drift/model-registry.ts:132`), under the existing `// Claude 5 families (text chat)` comment. Justification from the code, not from vibes: - `includeFamilies` is documented as "families aimock actually MOCKS … chat / text completion families" (`model-registry.ts:16-20`). Opus 5 is a GA text-chat/reasoning model on the live listing — squarely that category. - **Every** GA text-chat Claude family on the live listing is in `includeFamilies.anthropic`. `excludeFamilies.anthropic` holds only retired/legacy ids (`claude-v3`, `claude-2`, `claude-instant-1`, `model-registry.ts:217-222`). Opus 5 is brand new, not retired, so exclude would be wrong. - Direct sibling precedent: `claude-sonnet-5` and `claude-opus-4-8` are both `includeFamilies` entries added by `34056c2` ("INCLUDE (text chat / reasoning aimock mocks): … Anthropic all 4.x/5 point releases"). Opus 5 is the same class of thing. - It is not a preview and not Gemma, so neither exclude-by-rule predicate (`PREVIEW_FAMILY`, `GEMMA_FAMILY`) applies. ## Reasoning vs non-reasoning: reasoning-capable, and no code change needed `isReasoningModel("claude-opus-5")` **already returns `true`**, before and after this PR. Verified empirically (not reasoned) by driving the real `src/model-utils.ts` export on both tree states: ``` PRE-fix POST-fix claude-opus-5 -> true true claude-opus-5-20260724 -> true true claude-sonnet-5 -> true true claude-fable-5 -> true true claude-opus-4-8 -> true true claude-opus-4 -> true true claude-3-5-sonnet -> false false (control) claude-3-opus -> false false (control) ``` `true` is the **correct** verdict for an extended-thinking Opus-class model. It arrives via the documented fail-open default (`model-utils.ts:18-26`, `135-136`): no `NONREASONING_FAMILIES` prefix matches `claude-opus-5`, and `REASONING_FAMILIES`' `claude-opus-4` entry does not prefix-match `claude-opus-5` either, so it falls through to `true`. So **`src/model-utils.ts` is deliberately untouched.** Adding a `REASONING_FAMILIES` entry would be a no-op — that list is explicitly "mostly documentation; because unknown ids default to `true` it is not load-bearing for correctness" (`model-utils.ts:21-22`) — and it would break the precedent that `claude-sonnet-5` and `claude-fable-5` also rely on the default rather than an explicit entry. Touching shipped source for a zero-delta documentation edit would turn a registry-only change into a src change for no benefit. ## Mock / fixture support: none needed Registry classification alone is sufficient, and this PR does **not** claim support aimock lacks. aimock's Anthropic surface is fixture-driven and model-agnostic — it matches on request content, not on model id. Probed the built `dist/` with real HTTP requests against a running server: ``` POST /v1/messages {"model":"claude-opus-5", ...} -> {"error":{"message":"No fixture matched", ...}} POST /v1/messages {"model":"claude-sonnet-5", ...} -> {"error":{"message":"No fixture matched", ...}} ``` Byte-identical behavior to the already-included `claude-sonnet-5`: with a matching fixture loaded both serve it, without one both report `No fixture matched`. No per-family code path exists, so there is nothing family-specific to add. `DEFAULT_MODELS` (`src/server.ts:227`) is a separate static 5-id list that this registry does not feed and that no Claude-5 family was ever in. This is exactly the situation the existing `claude-fable-5` comment already documents — a family included ahead of a recorded fixture, intentionally. ## RED → GREEN The failure surface is the real live drift leg: `src/__tests__/drift/models.drift.ts:606-618`, `describe.skipIf(!process.env.ANTHROPIC_API_KEY)`, which calls `listAnthropicModels()` against the real API and asserts through `assertNoUnclassifiedFamilies`. **It gates on `ANTHROPIC_API_KEY` and is SKIPPED without one**, so reproducing the RED below requires credentials; a bare checkout silently skips it. A credential-free leg is included after. ### RED — unmodified `main` (d1b5304), live API ``` $ pnpm exec vitest run --config vitest.config.drift.ts \ src/__tests__/drift/models.drift.ts -t "live /models contains no unclassified family" × Anthropic Claude model-family availability > live /models contains no unclassified family 499ms → API DRIFT DETECTED: Anthropic Claude (live /models family canary) 1. [critical] Unclassified model family "claude-opus-5" in anthropic /models — add it to includeFamilies (aimock mocks it) or excludeFamilies (non-text / retired / preview) in model-registry.ts Path: models/claude-opus-5 SDK: (family in includeFamilies ∪ excludeFamilies) Real: claude-opus-5 Mock: <no mock leg — live /models family canary> : expected [ 'claude-opus-5' ] to deeply equal [] ❯ assertNoUnclassifiedFamilies src/__tests__/drift/models.drift.ts:95:32 ❯ src/__tests__/drift/models.drift.ts:611:7 Test Files 1 failed (1) Tests 1 failed | 24 skipped (25) ``` ### GREEN — same command, after the classification ``` $ pnpm exec vitest run --config vitest.config.drift.ts \ src/__tests__/drift/models.drift.ts -t "live /models contains no unclassified family" ✓ src/__tests__/drift/models.drift.ts (25 tests | 24 skipped) 506ms ✓ Anthropic Claude model-family availability > live /models contains no unclassified family 505ms Test Files 1 passed (1) Tests 1 passed | 24 skipped (25) ``` ### Credential-free corroboration Driving the pure mirror `unclassifiedFamiliesForSync` (`scripts/drift-sync.ts`) over the captured live payload, on both tree states: ``` captured LIVE anthropic /models ids (11): ["claude-opus-5","claude-sonnet-5","claude-fable-5", "claude-opus-4-8","claude-opus-4-7","claude-sonnet-4-6","claude-opus-4-6", "claude-opus-4-5-20251101","claude-haiku-4-5-20251001","claude-sonnet-4-5-20250929", "claude-opus-4-1-20250805"] PRE-fix : unclassifiedFamiliesForSync(anthropic) = ["claude-opus-5"] claude-opus-5 flagged = true includeFamilies.anthropic.has('claude-opus-5') = false POST-fix: unclassifiedFamiliesForSync(anthropic) = [] claude-opus-5 flagged = false includeFamilies.anthropic.has('claude-opus-5') = true excludeFamilies.anthropic.has('claude-opus-5') = false ``` ### Membership pin RED → GREEN The `DATA_FROZEN` canary correctly tripped on the deliberate addition: ``` × classification-data membership freeze > freezes includeFamilies.anthropic membership 5ms → Frozen data set "includeFamilies.anthropic" membership changed (now: ["claude-3-5-haiku", "claude-3-5-sonnet","claude-3-7-sonnet","claude-3-haiku","claude-3-opus","claude-3-sonnet", "claude-fable-5","claude-haiku-4","claude-haiku-4-5","claude-opus-4","claude-opus-4-1", "claude-opus-4-5","claude-opus-4-6","claude-opus-4-7","claude-opus-4-8","claude-opus-5", "claude-sonnet-4","claude-sonnet-4-5","claude-sonnet-4-6","claude-sonnet-5"]). If this is a deliberate, reviewed addition/removal of a classified family, update its pin here; if not, it is a silent canary-silencing edit and must be reverted. Expected: "dbd8b4ef9afd50057143480d89db373886e7c19abe36ef9b4421456305ca2509" Received: "ab79ff332fadeff93c2678ebe3e0af7a6280ce6f0deb4694228e316944dfeb74" Test Files 1 failed (1) Tests 1 failed | 19 passed (20) ``` The new pin was **derived independently**, not copied from vitest's `Received`. Reimplemented the test's own hashing (`sha256(JSON.stringify(members.sort()))`, `logic-pin.test.ts:59-61` + `:227`) in a standalone script, and first validated the method by reproducing the **pre-existing** pin exactly: ``` PRE derived : dbd8b4ef9afd50057143480d89db373886e7c19abe36ef9b4421456305ca2509 PRE in-repo : dbd8b4ef9afd50057143480d89db373886e7c19abe36ef9b4421456305ca2509 PRE METHOD VALIDATED: True POST derived : ab79ff332fadeff93c2678ebe3e0af7a6280ce6f0deb4694228e316944dfeb74 ``` Three-way agreement (derived / vitest `Received` / committed value). After re-pinning: ``` ✓ src/__tests__/drift/logic-pin.test.ts (20 tests) 3ms Test Files 1 passed (1) Tests 20 passed (20) ``` ## Full suite — zero regressions ``` $ pnpm test Test Files 166 passed (166) Tests 4768 passed | 3 skipped (4771) ``` ## Versioning verdict: no bump, no changeset **No behavior change**, so nothing to release. Three independent confirmations: 1. **No shipped importer.** Every importer of `model-registry.ts` is a test or a repo script (`scripts/drift-sync.ts`, `scripts/drift-sync-check.ts`, and six `src/__tests__/` files). Non-test, non-script importers: **none**. `src/model-utils.ts`, `src/server.ts` and the `tsdown` entrypoints never touch it — `includeFamilies`/`excludeFamilies` are drift metadata, not a runtime allowlist. 2. **Absent from the published tarball, and not inlined.** `package.json` `files` is an allowlist (`["dist","fixtures",".claude-plugin","skills","CHANGELOG.md"]`); `npm pack --dry-run` yields 576 files with **0** matching `__tests__` / `logic-pin` / `model-registry` / `src/`. `dist/` contains zero occurrences of `includeFamilies`, `excludeFamilies`, or `claude-opus-5`. 3. **Built artifact semantically identical.** Compared pre-fix and post-fix `dist/`, normalizing the two known non-reproducible-build artifacts this repo has (tsdown's import-alias numbering, e.g. `node_http0` vs `node_http1`, and declaration sourcemaps), with two identical-source builds as the noise floor: ``` noise floor (identical source): CONTROL: POST vs CTRL files=285 differing=0 setdiff=0 subject: SUBJECT: PRE-fix vs POST-fix files=285 differing=0 setdiff=0 VERDICT: PUBLISHED ARTIFACT SEMANTICALLY IDENTICAL ``` Release mechanism is a manual `package.json` version bump consumed by `publish-release.yml`; there is no Changesets / release-please / semantic-release, and no workflow derives a version from commit messages. `chore(drift):` is therefore the honest prefix — this is a classification-only change with zero behavior delta, so neither `feat` nor `fix` is warranted. ## Pre-push gates ``` $ pnpm format:check → All matched files use Prettier code style! $ pnpm lint → clean $ pnpm exec tsc --noEmit → clean $ pnpm test → 166 files passed, 4768 passed, 0 failed $ pnpm build → ✔ Build complete $ pnpm test:exports → node10 🟢 node16 CJS 🟢 node16 ESM 🟢 bundler 🟢 $ pnpm exec commitlint --from HEAD~1 --to HEAD → clean ``` ## Pre-existing caveat (not introduced here) `.github/workflows/unreleased-check.yml` counts commits touching `src/` since the last tag without excluding `src/__tests__/`, so on merge it will emit `::warning:: … version has not been bumped`. It is a warning with no `exit 1` and does not fail CI. #336 and #337 have the same shape. ## Live-CI RED → GREEN (added after CI ran) `drift-live-pr` runs the drift suite against the **real** provider `/models` listings and uploads a base and a head report. On this PR ([run 30132383648](https://github.com/CopilotKit/aimock/actions/runs/30132383648)) those two artifacts are themselves the red-green, from CI rather than from my machine: ``` BASE (main @ d1b5304) HEAD (this PR) "conclusion": "critical" "conclusion": "clean" entries: 1 "entries": [] - critical | claude-opus-5 ``` `main` is measurably red on this finding right now; this branch is clean. Full CI: **26 pass, 2 skipping, 0 fail** (28 rows).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Drift subsystem cleanup
Fast-follow to the deterministic self-healing drift re-architecture (#335). Four low-risk cleanups the CR flagged as follow-ups; no behavior change to the shipped remediation flow.
What changed
commitSyncChanges, which leftbuildPrBody,gatedCommitFiles,isProductionFile(and their orphaned helperstruncateBody/GH_BODY_MAX/GH_BODY_SAFE_MAX/affectedSkillSections/BUILDER_TO_SKILL_SECTION) with no production caller. Removed them and their tests.claude-fable-5— the registry intentionally carries a forward-looking family that isn't live yet, so the sync proposed removing it every run. Added a smallisForwardLookingFamilyallowlist applied at the sync-policy layer (detectDeprecatedFamiliesForSync); the canonical detector is unchanged, so genuine retirements are still detected.excludeFamilies/includeFamiliesdata —logic-pin.test.tspreviously froze only the classification rules, not the membership data; a family could have been silenced by adding it straight toexcludeFamilies. Added aDATA_FROZENchecksum block (hashes the live sets) that fails loudly on any membership change.detectDeprecatedFamiliesfor normal families, and explicitly asserts the one intended forward-looking divergence (so removing the filter above fails the test).checks: read/statuses: read(and their app-token counterparts + a stale comment) fromfix-drift.yml; nothing in the workflow consumes check-runs or statuses, and it does no auto-merge.Review
Tier-1 CR (correctness + test-guard quality), 2 reviewers, 0 mandatory findings — both empirically verified the new pins/guards are non-vacuous (recomputed all 6 data-pin hashes; confirmed the mirror-divergence test fails if the forward-looking filter is removed).
Test evidence
tsc --noEmit0 errors · full suite green ·logic-pin20 ·drift-sync-mirror-equivalence28.Deferred follow-ups (not in this PR)
$metadata: the 2 forced-credential failures in the bedrock drift leg are a category error in the drift test's own SDK-shape stubs — they compare the SDK client-output shape (which includes$metadata) against the raw wire body.$metadatais synthesized by the AWS SDK from HTTP status/headers after deserialization; it is not part of the Bedrock wire response. The correct fix is to the test stubs (dormant — needs a live Bedrock recording), not the mock.models.drift.ts, so a localpnpm testrun with provider API keys exported would also execute the live-availability suites (harmless in CI, which sets no keys). De-coupling would require extracting the canonical predicates into a non-.drift.tsmodule — deferred.🤖 Generated with Claude Code
https://claude.ai/code/session_01KamK73Wu5heLxJEogM6hHC