fix: classify gemini-3.5-flash-lite and gemini-3.6-flash families#322
Merged
Conversation
Google Gemini's live /models now returns two new text-generation families that the drift canary classified as neither included nor excluded, producing 2 critical drift diffs every daily run (CI run 29898337754): - gemini-3.5-flash-lite - gemini-3.6-flash aimock mocks Gemini text models, so both belong in includeFamilies.gemini (not excludeFamilies -- they are not preview/retired/non-text). Add them to includeFamilies AND to the static /models test wave in models.drift.ts so the offline test:drift suite catches this class of drift going forward (closing the fixture-vs-live gap, per #315).
commit: |
jpr5
added a commit
that referenced
this pull request
Jul 24, 2026
## Needs a human decision (drift-sync) The deterministic, zero-LLM drift-sync found a model-family change it must NOT auto-apply (a genuinely new/unclassified family, a still-referenced deprecation, or a registry structural mismatch). It wrote the note file(s) below and opened this PR so the decision is REACHABLE in the repo. > **Not auto-merged — a human decides.** > >⚠️ **The `Decision: include` instructions in this template do NOT apply to this > PR.** That protocol is scoped to a *new-family* note only: `renderProposalNote` > emits the `## Decision` block solely under `kind === "new-family"`, and > `parseProposalDecision` has exactly one call site — the addition half of > `runDriftSyncCore`. The deprecation half never reads a decision at all. > > Both notes in this PR are **still-referenced deprecations**, and they contain > **no `Decision:` line whatsoever**. Writing one would be a no-op. There is no > marker, of any value, that makes a later run mechanically resolve this note > class — only a human registry edit can. **That edit is now part of this PR** > (see *Human decision* at the bottom), so there is no two-run hand-off here: > merging this PR both records the decision and resolves the drift. > > (For reference, the original template text: for a *new-family* note you would > set `Decision: include` to classify it, or delete the note / close the PR to > reject, then merge — and the next run would apply the registry edit itself.) ### Note file(s) in this PR <!-- drift-changeset: e08d8fcc953752d5 --> - `drift-proposals/gemini-gemini-1.5-flash-deprecated-referenced.md` <!-- drift-proposal-note: drift-proposals/gemini-gemini-1.5-flash-deprecated-referenced.md --> - `drift-proposals/gemini-gemini-1.5-pro-deprecated-referenced.md` <!-- drift-proposal-note: drift-proposals/gemini-gemini-1.5-pro-deprecated-referenced.md --> ### drift-sync outcome ``` npm warn Unknown project config "minimum-release-age". This will stop working in the next major version of npm. See `npm help npmrc` for supported config options. npm warn Unknown project config "block-exotic-subdeps". This will stop working in the next major version of npm. See `npm help npmrc` for supported config options. needs-human note(s) written this run — routed to human without a live re-collect (no registry edit to re-verify) [needs-human-still-referenced] gemini/gemini-1.5-flash: "gemini-1.5-flash" is deprecated but still referenced in source — routed to human (drift-proposals/gemini-gemini-1.5-flash-deprecated-referenced.md) [needs-human-still-referenced] gemini/gemini-1.5-pro: "gemini-1.5-pro" is deprecated but still referenced in source — routed to human (drift-proposals/gemini-gemini-1.5-pro-deprecated-referenced.md) [skipped] anthropic: live /models listing too short to trust for anthropic (10 raw id(s), need >= 19 — the number of families aimock mocks for this provider) — never mass-removing off a truncated or empty listing npm warn Unknown env config "block-exotic-subdeps". This will stop working in the next major version of npm. See `npm help npmrc` for supported config options. npm warn Unknown env config "minimum-release-age". This will stop working in the next major version of npm. See `npm help npmrc` for supported config options. npm warn Unknown project config "minimum-release-age". This will stop working in the next major version of npm. See `npm help npmrc` for supported config options. npm warn Unknown project config "block-exotic-subdeps". This will stop working in the next major version of npm. See `npm help npmrc` for supported config options. ^[[33m[STARTED]^[[39m Backing up original state... ^[[32m[COMPLETED]^[[39m Backed up original state in git stash (c62c488) ^[[33m[STARTED]^[[39m Running tasks for staged files... ^[[33m[STARTED]^[[39m package.json^[[0;90m — 2 files^[[0m ^[[33m[STARTED]^[[39m *.{ts,mts,js,mjs,cjs,json,html,css,md}^[[0;90m — 2 files^[[0m ^[[33m[STARTED]^[[39m *.{ts,mts,js,mjs}^[[0;90m — 0 files^[[0m ^[[33m[SKIPPED]^[[39m *.{ts,mts,js,mjs}^[[0;90m — no files^[[0m ^[[33m[STARTED]^[[39m prettier --write ^[[32m[COMPLETED]^[[39m prettier --write ^[[32m[COMPLETED]^[[39m *.{ts,mts,js,mjs,cjs,json,html,css,md}^[[0;90m — 2 files^[[0m ^[[32m[COMPLETED]^[[39m package.json^[[0;90m — 2 files^[[0m ^[[32m[COMPLETED]^[[39m Running tasks for staged files... ^[[33m[STARTED]^[[39m Applying modifications from tasks... ^[[32m[COMPLETED]^[[39m Applying modifications from tasks... ^[[33m[STARTED]^[[39m Cleaning up temporary files... ^[[32m[COMPLETED]^[[39m Cleaning up temporary files... [fix/drift-2026-07-24-30074183288 85fc683] fix(drift-sync): mechanical model-family sync (needs-human note file(s)) 2 files changed, 14 insertions(+) create mode 100644 drift-proposals/gemini-gemini-1.5-flash-deprecated-referenced.md create mode 100644 drift-proposals/gemini-gemini-1.5-pro-deprecated-referenced.md reason=needs-human changeset-key=e08d8fcc953752d5 ``` --- ## Human decision: keep mocking these families **Decision: KEEP.** `gemini-1.5-pro` and `gemini-1.5-flash` are gone from Gemini's live `/models` listing, but aimock still mocks them on purpose — clients pinned to older SDKs and older CopilotKit configs still send those ids, and `isReasoningModel()` must keep answering `false` for them. So nothing is being removed. Instead, commit `a3dc250` records the decision in the registry by moving both families from `includeFamilies.gemini` into `excludeFamilies.gemini`'s existing **"Retired / legacy specialty"** bucket, and re-pins the two affected `DATA_FROZEN` membership hashes. `excludeFamilies` means *"not counted as text-generation **drift**"*, not *"not mocked"* — `gemini-pro`, `gpt-3.5` and `gemini-2.0-flash-exp` already live there while being mocked and heavily referenced in `src/`. Prior art for exactly this move: #315 (`fix(drift): classify bare chat-latest alias in openai excludeFamilies`). Because `detectDeprecatedFamiliesForSync` draws its candidate set from `includeFamilies[provider]`, the two families can no longer become deprecation candidates — drift-sync stops re-flagging them needs-human, which ends the daily red job and the daily Slack page. The two note files stay in this PR as the audit trail for the `excludeFamilies` entries; they become inert on merge and will never be re-read or re-written. **Nothing that references `gemini-1.5-*` was touched.** `src/model-utils.ts:54` (`NONREASONING_FAMILIES`), the three gemini reasoning-capability suites, the `gemini.test.ts` / `vertex-ai.test.ts` conformance assertions and the recorded `sdk-shapes.ts` fixture are all unchanged. ### Diff ``` src/__tests__/drift/logic-pin.test.ts | 4 ++-- src/__tests__/drift/model-registry.ts | 12 +++++++++--- 2 files changed, 11 insertions(+), 5 deletions(-) ``` ### Known side effect Gemini's truncated-listing trust floor is `includeFamilies[provider].size`, so it drops from 11 to 9. Reversible: move the families back and re-pin. --- ## Local red-green proof ### 1. Baseline — before any edit, drift suites all green ``` $ pnpm exec vitest run src/__tests__/drift/ ✓ src/__tests__/drift/model-registry.ts ... (all 9 files) Test Files 9 passed (9) Tests 87 passed (87) ``` ### 2. RED — registry membership moved, pins NOT yet updated `logic-pin.test.ts` fails loudly on the stale membership pins, exactly as its module doc promises: ``` $ pnpm exec vitest run src/__tests__/drift/logic-pin.test.ts × classification-data membership freeze > freezes includeFamilies.gemini membership → Frozen data set "includeFamilies.gemini" membership changed (now: ["gemini-2.0-flash","gemini-2.0-flash-lite","gemini-2.5-flash", "gemini-2.5-flash-lite","gemini-2.5-pro","gemini-3.1-flash-lite", "gemini-3.5-flash","gemini-3.5-flash-lite","gemini-3.6-flash"]). Expected: "4a9428b64ffcff0fbb79878d88ed993ffac53e43d64e26bcf5d86509626f593d" Received: "c2e2c56b8f8d5fc56152b4633e7d3782e95b7eeb9bc123da71f00e884a54a743" × classification-data membership freeze > freezes excludeFamilies.gemini membership → Frozen data set "excludeFamilies.gemini" membership changed (now: ["aqa","gemini-1.5-flash","gemini-1.5-pro","gemini-2.0-flash-exp", ...]). Expected: "e3545138234ad782937f66760bef942fca7d4bd0934a87da30bf6e5816ba69b1" Received: "c95dedab7588212bbba0bf9ab6434bfd43a449adbe7b35f81f48271ee849a9c2" Test Files 1 failed (1) Tests 2 failed | 18 passed (20) ``` The two new pin values were derived independently — `sha256(JSON.stringify([...set].sort()))` recomputed from `model-registry.ts` in a standalone script, matching the values the test itself reports: ``` includeFamilies.gemini c2e2c56b8f8d5fc56152b4633e7d3782e95b7eeb9bc123da71f00e884a54a743 excludeFamilies.gemini c95dedab7588212bbba0bf9ab6434bfd43a449adbe7b35f81f48271ee849a9c2 ``` ### 3. GREEN — pins re-pinned, same command ``` $ pnpm exec vitest run src/__tests__/drift/logic-pin.test.ts ✓ src/__tests__/drift/logic-pin.test.ts (20 tests) 3ms Test Files 1 passed (1) Tests 20 passed (20) ``` ### 4. Full suite — zero regressions ``` $ pnpm test Test Files 165 passed | 1 skipped (166) Tests 4724 passed | 47 skipped (4771) ``` And the suites that specifically pin `gemini-1.5-*` behavior: ``` $ pnpm exec vitest run src/__tests__/model-utils.test.ts \ src/__tests__/gemini-reasoning-capability.test.ts \ src/__tests__/gemini-audio-reasoning-capability.test.ts \ src/__tests__/gemini-toolonly-reasoning-capability.test.ts \ src/__tests__/gemini.test.ts src/__tests__/vertex-ai.test.ts \ src/__tests__/drift/ src/__tests__/drift-sync-core.test.ts Test Files 16 passed (16) Tests 236 passed (236) ``` ### 5. Drift resolution proved offline `scripts/drift-sync.ts` has no `--dry-run` / `--check` flag, so the canary was exercised directly instead: `detectDeprecatedFamiliesForSync` is exported and pure, so it was called against a simulated live `/models` listing containing every gemini family aimock classifies **except** `gemini-1.5-pro` / `gemini-1.5-flash` (31 ids — comfortably over the trust floor under both the old size 11 and the new size 9), once on the pre-fix tree and once on the fixed tree. Pre-fix (this branch at `85fc683`) — reproduces the needs-human routing: ``` includeFamilies.gemini.size (trust floor) = 11 simulated live listing length = 31 status = checked candidates = [{"provider":"gemini","family":"gemini-1.5-flash","stillReferenced":true}, {"provider":"gemini","family":"gemini-1.5-pro","stillReferenced":true}] gemini-1.5 candidates = 2 needs-human candidates = 2 >>> STILL FLAGGED (red) ``` Post-fix (`a3dc250`) — resolved: ``` includeFamilies.gemini.size (trust floor) = 9 simulated live listing length = 31 status = checked candidates = [] gemini-1.5 candidates = 0 needs-human candidates = 0 >>> RESOLVED: gemini-1.5 NOT flagged ``` `detectDeprecatedFamiliesForSync` is the exact function `drift-sync.ts` uses to build the deprecation candidate set, so this is the real routing surface, not a stand-in. **Plus a genuine LIVE confirmation from this PR's own CI.** The `drift-live-pr` check ran the drift suite against the real provider `/models` listings on this head commit (`preflight: Gemini key OK`) and the uploaded head drift report contains exactly one entry — a pre-existing *anthropic* `claude-opus-5` unclassified-family finding, also present in the base report — with **zero** `gemini-1.5` mentions and no gemini entry at all: ``` Drift gate PASSED: 1 advisory (pre-existing) finding(s) surfaced, none new-in-head. ``` That confirms against the live listing that moving both families to `excludeFamilies` introduced no new drift and no unclassified-family regression (`isClassifiedFamily` is satisfied by include **or** exclude). To be precise about what it does *not* prove: this leg does not surface the still-referenced-deprecation class as a report entry — the base report shows zero `gemini-1.5` mentions too — so the live job is regression evidence, and the needs-human routing fix itself is proved by the `detectDeprecatedFamiliesForSync` red-green above. ### 6. Pre-push gates ``` $ pnpm format:check → All matched files use Prettier code style! $ pnpm lint → clean (no output) $ pnpm exec tsc --noEmit → clean (no output) $ pnpm build → ✔ Build complete (273 files) $ pnpm test:exports → No problems found 🌟 ``` ### 7. Zero behavior change → no version bump, no changeset The test that matters is not "which directory did the files sit in" but **does this change aimock's behavior for a consumer?** It does not, and that is verified three independent ways. **(a) The registry is not reachable from shipped code.** Every importer of `src/__tests__/drift/model-registry.ts` is a test or the maintenance script: ``` scripts/drift-sync.ts (repo script, not published) src/__tests__/drift-sync-core.test.ts src/__tests__/drift-sync-mirror-equivalence.test.ts src/__tests__/drift/logic-pin.test.ts src/__tests__/drift/model-registry.test.ts src/__tests__/drift/models.drift.ts ``` No shipped-source importer exists — `src/model-utils.ts`, `src/server.ts`, the handlers and every published entrypoint in `tsdown.config.ts` are all absent from that list. `includeFamilies` / `excludeFamilies` are drift-detection *metadata*, not a runtime allowlist. Confirmed at the artifact level too: `dist/` contains zero occurrences of `includeFamilies`, `excludeFamilies` or `isClassifiedFamily`, so the data isn't inlined either, and `npm pack --dry-run` ships 576 files with **none** matching `__tests__` / `/drift/` / `model-registry` / `logic-pin` / `src/` (`files` allowlist is `["dist","fixtures",".claude-plugin","skills","CHANGELOG.md"]`; `scripts/` ships 0 entries; no `.npmignore`). **(b) The published artifact is byte-identical.** Built `dist/` from the pre-fix and post-fix trees and compared all 285 shipped code/declaration files. Note the build is **not** reproducible — two builds of *identical* source differ in declaration sourcemaps and in tsdown's non-deterministic import-alias numbering (`node_http0` ↔ `node_http1`), so a naive whole-dist hash gives a false positive. Normalizing that noise, with two identical-source builds as the control noise floor: ``` control: two IDENTICAL-source builds → files=285 differing=0 subject: PRE-fix vs POST-fix build → files=285 differing=0 VERDICT: PUBLISHED ARTIFACT SEMANTICALLY IDENTICAL ``` **(c) Consumer-visible behavior is bit-identical.** Probed the *built* `dist/` before and after with real requests for both ids — `generateContent`, SSE `streamGenerateContent`, strict-on and strict-off reasoning, the OpenAI-compatible `/v1/chat/completions` surface, and `/v1/models` — plus the `isReasoningModel()` verdicts and the captured capability warnings. The two JSON captures compare **equal**: ``` PRE isReasoningModel: {'gemini-1.5-pro': False, 'gemini-1.5-flash': False} POST isReasoningModel: {'gemini-1.5-pro': False, 'gemini-1.5-flash': False} controls (unchanged): gemini-2.5-pro=True, gemini-pro=True, some-future-model=True 8/8 request probes identical (all HTTP 200, identical bodies) capability warnings identical, still naming both gemini-1.5 ids identical overall: True ``` So aimock still mocks both ids identically, `isReasoningModel()` still returns `false` for both, and the advertised model list is unchanged. **Release mechanism.** `publish-release.yml` triggers on push to `main`, compares `package.json`'s version against npm, and publishes + tags only when that version is not yet published — i.e. releases come from a **manual `package.json` version bump**. There is no Changesets setup (no `.changeset/`, no `@changesets/cli`), no release-please and no semantic-release, and no workflow derives a version from commit messages, so **the conventional-commit type does not trigger a release**; the `fix(drift):` prefix here matches the repo's own prior art for registry classification changes (#315, #322) and is inert with respect to versioning. `publish-commit.yml` → `pkg-pr-new` is just the per-PR preview publish behind the "Continuous Releases" / "preview" checks and gates nothing on versioning. **No changeset file and no version bump are required.** Minor pre-existing caveat: `unreleased-check.yml` counts commits touching `src/` since the last tag without excluding `src/__tests__/`, so on merge it will emit a `::warning::` that the version wasn't bumped. It is a warning only — no `exit 1` — so it does not fail CI. #336 had the same shape. --- ## Separate latent defect, filed for follow-up (NOT introduced by this PR) The *mechanical* registry-edit path documented for new-family notes is currently dead-locked by its own gate, so `ok-applied` can never happen: gate-2 of `drift-sync-check.ts` re-runs `logic-pin.test.ts` as a subprocess, that file's `DATA_FROZEN` block freezes set **membership**, and a mechanical add/remove *is* a membership change — so it trips the pin, gets reverted by `revertFiles`, and reports `gate-failed`. #335 shipped the mechanical path; #336 added `DATA_FROZEN` 40 minutes later. It has never been observed because no `ok-applied` run has ever occurred. The `ok-applied` test injects a fake gate (`makeFakeDeps`), so it asserts `runSyncCheck` was *called*, never that the real pin would pass. Worth fixing separately — either exempt the `DATA_FROZEN` membership hashes from gate-2, or have drift-sync re-pin the hash as part of the allowlisted data edit and lean on human PR review as the guard.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Google Gemini's live
GET /modelsnow returns two new text-generation families that aimock's drift canary classified as neither included nor excluded, producing 2 critical drift diffs on every daily run:gemini-3.5-flash-litegemini-3.6-flashaimock mocks Gemini text models, so both belong in
includeFamilies.gemini(notexcludeFamilies— they are not preview / retired / non-text). This adds them toincludeFamiliesand to the static/modelstest wave inmodels.drift.ts, so the offlinetest:driftsuite catches this class of drift going forward — closing the fixture-vs-live gap (per #315).This touches only
src/__tests__/drift/*. It does not ship in the published npm package, so there is no version bump and no CHANGELOG entry.Local red-green proof
The real failure surface is the drift classifier (
unclassifiedFamilies) exercised by the static/modelstest wave. Both legs use the same command:pnpm run test:drift.RED — add the two families to the static
/modelswave snapshot only (what live now returns), WITHOUT classifying them:GREEN — add both families to
includeFamilies.geminiinmodel-registry.ts, re-run the same command:Full suite (
pnpm test) andpnpm run buildboth green afterward.Live corroboration
This drift was also live-red/green-verified in CI run 29898337754: the pre-fix collector exited 2 with these exact 2 diffs; a post-fix re-collect exited 0. The local red-green above is the required proof; the CI run is corroborating live evidence.
Scope
This is the tactical half of a two-part fix. The other half reconciles the drift-success-predicate so the bot can auto-classify canary drift going forward.