Skip to content

chore(drift): classify claude-opus-5 in includeFamilies (live GA text chat)#339

Merged
jpr5 merged 1 commit into
mainfrom
drift/classify-claude-opus-5
Jul 24, 2026
Merged

chore(drift): classify claude-opus-5 in includeFamilies (live GA text chat)#339
jpr5 merged 1 commit into
mainfrom
drift/classify-claude-opus-5

Conversation

@jpr5

@jpr5 jpr5 commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

What was unclassified

claude-opus-5Claude Opus 5 — is an unclassified Anthropic model family on main. It appears in Anthropic's live GET /v1/models listing but is in neither includeFamilies.anthropic nor excludeFamilies.anthropic, so the live /models family canary flags it as a new-family decision and re-routes it to a human on every run.

It was simply missed: the last classification wave (34056c2) covered the 2026-07-16 listing, and Opus 5 landed after it (created_at: 2026-07-24).

Only this one family is unclassified. The live listing returns 11 ids and every other one is already classified — including the other two Claude-5 members, claude-sonnet-5 and claude-fable-5. So this is a one-family fix, not a wave.

Evidence it was unclassified

Live listing (GET https://api.anthropic.com/v1/models, fetched 2026-07-24, key sourced from the session secret cache — never persisted anywhere):

count 11  has_more False
  claude-opus-5              | Claude Opus 5    | 2026-07-24T00:00:00Z   <-- unclassified
  claude-sonnet-5            | Claude Sonnet 5  | 2026-06-29T00:00:00Z
  claude-fable-5             | Claude Fable 5   | 2026-06-07T00:00:00Z
  claude-opus-4-8            | Claude Opus 4.8  | 2026-05-28T00:00:00Z
  claude-opus-4-7            | Claude Opus 4.7  | 2026-04-14T00:00:00Z
  claude-sonnet-4-6          | Claude Sonnet 4.6| 2026-02-17T00:00:00Z
  claude-opus-4-6            | Claude Opus 4.6  | 2026-02-04T00:00:00Z
  claude-opus-4-5-20251101   | Claude Opus 4.5  | 2025-11-24T00:00:00Z
  claude-haiku-4-5-20251001  | Claude Haiku 4.5 | 2025-10-15T00:00:00Z
  claude-sonnet-4-5-20250929 | Claude Sonnet 4.5| 2025-09-29T00:00:00Z
  claude-opus-4-1-20250805   | Claude Opus 4.1  | 2025-08-05T00:00:00Z

Independently corroborated by PR #337's drift-live-pr job, whose uploaded head and base drift reports both contain exactly one entry — this same finding — with conclusion: critical. It passed #337's delta gate only because that gate is new-in-head-only, so the finding has been sitting on main unaddressed.

Classification chosen, and why

includeFamilies.anthropic (src/__tests__/drift/model-registry.ts:132), under the existing // Claude 5 families (text chat) comment.

Justification from the code, not from vibes:

  • includeFamilies is documented as "families aimock actually MOCKS … chat / text completion families" (model-registry.ts:16-20). Opus 5 is a GA text-chat/reasoning model on the live listing — squarely that category.
  • Every GA text-chat Claude family on the live listing is in includeFamilies.anthropic. excludeFamilies.anthropic holds only retired/legacy ids (claude-v3, claude-2, claude-instant-1, model-registry.ts:217-222). Opus 5 is brand new, not retired, so exclude would be wrong.
  • Direct sibling precedent: claude-sonnet-5 and claude-opus-4-8 are both includeFamilies entries added by 34056c2 ("INCLUDE (text chat / reasoning aimock mocks): … Anthropic all 4.x/5 point releases"). Opus 5 is the same class of thing.
  • It is not a preview and not Gemma, so neither exclude-by-rule predicate (PREVIEW_FAMILY, GEMMA_FAMILY) applies.

Reasoning vs non-reasoning: reasoning-capable, and no code change needed

isReasoningModel("claude-opus-5") already returns true, before and after this PR. Verified empirically (not reasoned) by driving the real src/model-utils.ts export on both tree states:

                            PRE-fix   POST-fix
  claude-opus-5           ->  true      true
  claude-opus-5-20260724  ->  true      true
  claude-sonnet-5         ->  true      true
  claude-fable-5          ->  true      true
  claude-opus-4-8         ->  true      true
  claude-opus-4           ->  true      true
  claude-3-5-sonnet       ->  false     false     (control)
  claude-3-opus           ->  false     false     (control)

true is the correct verdict for an extended-thinking Opus-class model. It arrives via the documented fail-open default (model-utils.ts:18-26, 135-136): no NONREASONING_FAMILIES prefix matches claude-opus-5, and REASONING_FAMILIES' claude-opus-4 entry does not prefix-match claude-opus-5 either, so it falls through to true.

So src/model-utils.ts is deliberately untouched. Adding a REASONING_FAMILIES entry would be a no-op — that list is explicitly "mostly documentation; because unknown ids default to true it is not load-bearing for correctness" (model-utils.ts:21-22) — and it would break the precedent that claude-sonnet-5 and claude-fable-5 also rely on the default rather than an explicit entry. Touching shipped source for a zero-delta documentation edit would turn a registry-only change into a src change for no benefit.

Mock / fixture support: none needed

Registry classification alone is sufficient, and this PR does not claim support aimock lacks.

aimock's Anthropic surface is fixture-driven and model-agnostic — it matches on request content, not on model id. Probed the built dist/ with real HTTP requests against a running server:

POST /v1/messages {"model":"claude-opus-5",   ...}  -> {"error":{"message":"No fixture matched", ...}}
POST /v1/messages {"model":"claude-sonnet-5", ...}  -> {"error":{"message":"No fixture matched", ...}}

Byte-identical behavior to the already-included claude-sonnet-5: with a matching fixture loaded both serve it, without one both report No fixture matched. No per-family code path exists, so there is nothing family-specific to add. DEFAULT_MODELS (src/server.ts:227) is a separate static 5-id list that this registry does not feed and that no Claude-5 family was ever in.

This is exactly the situation the existing claude-fable-5 comment already documents — a family included ahead of a recorded fixture, intentionally.

RED → GREEN

The failure surface is the real live drift leg: src/__tests__/drift/models.drift.ts:606-618, describe.skipIf(!process.env.ANTHROPIC_API_KEY), which calls listAnthropicModels() against the real API and asserts through assertNoUnclassifiedFamilies. It gates on ANTHROPIC_API_KEY and is SKIPPED without one, so reproducing the RED below requires credentials; a bare checkout silently skips it. A credential-free leg is included after.

RED — unmodified main (d1b5304), live API

$ pnpm exec vitest run --config vitest.config.drift.ts \
    src/__tests__/drift/models.drift.ts -t "live /models contains no unclassified family"

 × Anthropic Claude model-family availability > live /models contains no unclassified family 499ms
   →
API DRIFT DETECTED: Anthropic Claude (live /models family canary)

  1. [critical] Unclassified model family "claude-opus-5" in anthropic /models — add it to includeFamilies (aimock mocks it) or excludeFamilies (non-text / retired / preview) in model-registry.ts
     Path:    models/claude-opus-5
     SDK:     (family in includeFamilies ∪ excludeFamilies)
     Real:    claude-opus-5
     Mock:    <no mock leg — live /models family canary>
: expected [ 'claude-opus-5' ] to deeply equal []

 ❯ assertNoUnclassifiedFamilies src/__tests__/drift/models.drift.ts:95:32
 ❯ src/__tests__/drift/models.drift.ts:611:7

 Test Files  1 failed (1)
      Tests  1 failed | 24 skipped (25)

GREEN — same command, after the classification

$ pnpm exec vitest run --config vitest.config.drift.ts \
    src/__tests__/drift/models.drift.ts -t "live /models contains no unclassified family"

 ✓ src/__tests__/drift/models.drift.ts (25 tests | 24 skipped) 506ms
   ✓ Anthropic Claude model-family availability > live /models contains no unclassified family  505ms

 Test Files  1 passed (1)
      Tests  1 passed | 24 skipped (25)

Credential-free corroboration

Driving the pure mirror unclassifiedFamiliesForSync (scripts/drift-sync.ts) over the captured live payload, on both tree states:

captured LIVE anthropic /models ids (11): ["claude-opus-5","claude-sonnet-5","claude-fable-5",
  "claude-opus-4-8","claude-opus-4-7","claude-sonnet-4-6","claude-opus-4-6",
  "claude-opus-4-5-20251101","claude-haiku-4-5-20251001","claude-sonnet-4-5-20250929",
  "claude-opus-4-1-20250805"]

PRE-fix : unclassifiedFamiliesForSync(anthropic) = ["claude-opus-5"]   claude-opus-5 flagged = true
          includeFamilies.anthropic.has('claude-opus-5') = false
POST-fix: unclassifiedFamiliesForSync(anthropic) = []                  claude-opus-5 flagged = false
          includeFamilies.anthropic.has('claude-opus-5') = true
          excludeFamilies.anthropic.has('claude-opus-5') = false

Membership pin RED → GREEN

The DATA_FROZEN canary correctly tripped on the deliberate addition:

 × classification-data membership freeze > freezes includeFamilies.anthropic membership 5ms
   → Frozen data set "includeFamilies.anthropic" membership changed (now: ["claude-3-5-haiku",
     "claude-3-5-sonnet","claude-3-7-sonnet","claude-3-haiku","claude-3-opus","claude-3-sonnet",
     "claude-fable-5","claude-haiku-4","claude-haiku-4-5","claude-opus-4","claude-opus-4-1",
     "claude-opus-4-5","claude-opus-4-6","claude-opus-4-7","claude-opus-4-8","claude-opus-5",
     "claude-sonnet-4","claude-sonnet-4-5","claude-sonnet-4-6","claude-sonnet-5"]). If this is a
     deliberate, reviewed addition/removal of a classified family, update its pin here; if not,
     it is a silent canary-silencing edit and must be reverted.
     Expected: "dbd8b4ef9afd50057143480d89db373886e7c19abe36ef9b4421456305ca2509"
     Received: "ab79ff332fadeff93c2678ebe3e0af7a6280ce6f0deb4694228e316944dfeb74"

 Test Files  1 failed (1)
      Tests  1 failed | 19 passed (20)

The new pin was derived independently, not copied from vitest's Received. Reimplemented the test's own hashing (sha256(JSON.stringify(members.sort())), logic-pin.test.ts:59-61 + :227) in a standalone script, and first validated the method by reproducing the pre-existing pin exactly:

PRE  derived : dbd8b4ef9afd50057143480d89db373886e7c19abe36ef9b4421456305ca2509
PRE  in-repo : dbd8b4ef9afd50057143480d89db373886e7c19abe36ef9b4421456305ca2509
PRE  METHOD VALIDATED: True
POST derived : ab79ff332fadeff93c2678ebe3e0af7a6280ce6f0deb4694228e316944dfeb74

Three-way agreement (derived / vitest Received / committed value). After re-pinning:

 ✓ src/__tests__/drift/logic-pin.test.ts (20 tests) 3ms
 Test Files  1 passed (1)
      Tests  20 passed (20)

Full suite — zero regressions

$ pnpm test
 Test Files  166 passed (166)
      Tests  4768 passed | 3 skipped (4771)

Versioning verdict: no bump, no changeset

No behavior change, so nothing to release. Three independent confirmations:

  1. No shipped importer. Every importer of model-registry.ts is a test or a repo script (scripts/drift-sync.ts, scripts/drift-sync-check.ts, and six src/__tests__/ files). Non-test, non-script importers: none. src/model-utils.ts, src/server.ts and the tsdown entrypoints never touch it — includeFamilies/excludeFamilies are drift metadata, not a runtime allowlist.
  2. Absent from the published tarball, and not inlined. package.json files is an allowlist (["dist","fixtures",".claude-plugin","skills","CHANGELOG.md"]); npm pack --dry-run yields 576 files with 0 matching __tests__ / logic-pin / model-registry / src/. dist/ contains zero occurrences of includeFamilies, excludeFamilies, or claude-opus-5.
  3. Built artifact semantically identical. Compared pre-fix and post-fix dist/, normalizing the two known non-reproducible-build artifacts this repo has (tsdown's import-alias numbering, e.g. node_http0 vs node_http1, and declaration sourcemaps), with two identical-source builds as the noise floor:
noise floor (identical source):
  CONTROL: POST vs CTRL      files=285 differing=0 setdiff=0
subject:
  SUBJECT: PRE-fix vs POST-fix   files=285 differing=0 setdiff=0
VERDICT: PUBLISHED ARTIFACT SEMANTICALLY IDENTICAL

Release mechanism is a manual package.json version bump consumed by publish-release.yml; there is no Changesets / release-please / semantic-release, and no workflow derives a version from commit messages. chore(drift): is therefore the honest prefix — this is a classification-only change with zero behavior delta, so neither feat nor fix is warranted.

Pre-push gates

$ pnpm format:check      → All matched files use Prettier code style!
$ pnpm lint              → clean
$ pnpm exec tsc --noEmit → clean
$ pnpm test              → 166 files passed, 4768 passed, 0 failed
$ pnpm build             → ✔ Build complete
$ pnpm test:exports      → node10 🟢  node16 CJS 🟢  node16 ESM 🟢  bundler 🟢
$ pnpm exec commitlint --from HEAD~1 --to HEAD → clean

Pre-existing caveat (not introduced here)

.github/workflows/unreleased-check.yml counts commits touching src/ since the last tag without excluding src/__tests__/, so on merge it will emit ::warning:: … version has not been bumped. It is a warning with no exit 1 and does not fail CI. #336 and #337 have the same shape.

Live-CI RED → GREEN (added after CI ran)

drift-live-pr runs the drift suite against the real provider /models listings and uploads a base and a head report. On this PR (run 30132383648) those two artifacts are themselves the red-green, from CI rather than from my machine:

BASE (main @ d1b5304)                    HEAD (this PR)
  "conclusion": "critical"                 "conclusion": "clean"
  entries: 1                               "entries": []
    - critical | claude-opus-5

main is measurably red on this finding right now; this branch is clean. Full CI: 26 pass, 2 skipping, 0 fail (28 rows).

… chat)

The live /models family canary is red on main: Anthropic shipped
`claude-opus-5` (Claude Opus 5, created_at 2026-07-24) and the registry
has no classification for it, so `unclassifiedFamilies` flags it as a
new-family decision and keeps routing it to a human every run.

Classify it as INCLUDE — it is a GA text-chat/reasoning family on the
live listing, exactly like its already-included siblings
`claude-sonnet-5` and `claude-opus-4-8`. All 11 ids the live Anthropic
listing returns are text-chat Claude families and every other one is
already in includeFamilies.anthropic; opus-5 was simply missed because it
landed after the last classification wave (34056c2).

No reasoning-map change. `isReasoningModel("claude-opus-5")` already
returns true via the documented fail-open default (no NONREASONING_FAMILIES
prefix matches), which is the correct verdict for an extended-thinking
Opus-class model. That matches the precedent set for claude-sonnet-5 and
claude-fable-5, which also rely on the default rather than a
REASONING_FAMILIES entry, so src/model-utils.ts is untouched.

No mock/fixture work needed: aimock's Anthropic surface is fixture-driven
and model-agnostic, so claude-opus-5 already behaves identically to the
included claude-sonnet-5.

Re-pin includeFamilies.anthropic membership — the DATA_FROZEN canary
correctly trips on the deliberate addition. New pin derived independently
from the test's own sha256(JSON.stringify(sorted)) and cross-checked
against vitest's reported value.

Registry data is test/script-only (no shipped importer, absent from the
npm tarball, dist/ semantically identical pre/post), so this carries zero
behavior delta and no version bump.
@pkg-pr-new

pkg-pr-new Bot commented Jul 24, 2026

Copy link
Copy Markdown

Open in StackBlitz

npm i https://pkg.pr.new/@copilotkit/aimock@339

commit: 72f85f8

@jpr5
jpr5 merged commit f988b1c into main Jul 24, 2026
28 checks passed
@jpr5
jpr5 deleted the drift/classify-claude-opus-5 branch July 24, 2026 23:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant