OpenRouter chat/LLM router simulation#328
Merged
Merged
Conversation
Detect OpenRouter by the canonical /api/v1 base prefix and shape the
OpenAI-compatible chat response into OpenRouter form: gen- id, top-level
provider (winner slug-author default, fixture-overridable), finish_reason
+ native_finish_reason, always-present system_fingerprint/service_tier, and
rich fixture-scriptable usage (cost, three-field cost_details, is_byok,
token-detail objects). Field shapes verified against real captured bytes.
models[] fallback simulation: iterate [model, ...models], serve the first
non-error match, fall through 429/503 runtime-error candidates, honor
provider.allow_fallbacks:false (fail-closed), echo the winning slug as
response.model. Discovery GET /api/v1/models|key|credits. OpenRouter error
envelope {error:{code,message,metadata?}} (code==status). Opt-in
: OPENROUTER PROCESSING SSE keepalive. Usage estimation unified across
streaming/non-streaming. OpenAI /v1 callers byte-for-byte unchanged.
Adds a FEATURE_RULES entry so the competitive-matrix drift job tracks
whether any competitor ships OpenRouter router mocking. 20+ tests incl.
OpenAI-streaming byte-isolation and tool-call shaping.
New docs/openrouter-chat page (examples from real captured bytes), sidebar/ landing/compatible-providers/fixtures/record-replay/migrate pages, README (13 providers / 15 surfaces), write-fixtures skill, CHANGELOG. Adds the 'OpenRouter router / fallback simulation' row to the competitive matrix (aimock deterministic; competitors none).
commit: |
jpr5
marked this pull request as ready for review
July 23, 2026 00:44
jpr5
added a commit
that referenced
this pull request
Jul 23, 2026
## Error-class-bound fallthrough for the OpenRouter `models[]` fallback engine Follow-up to #328. That PR shipped `models[]` failover (an error-fixture candidate falls through to the next model, unless `provider.allow_fallbacks:false`). But real OpenRouter fails over **inconsistently by error class** — the loudest authentic user complaint (openclaw#60191, #45834, #19249): a `403 budget-exceeded` or generic "provider error" does **not** advance to the next model (it retries the same primary), while 429/503 do. Users can't rehearse that. ### What this adds An optional **`fallthrough?: boolean`** on the error fixture response: - **absent (default)** → current behavior: the error candidate falls through to the next `models[]` candidate (fully backward-compatible). - **`fallthrough: false`** → the error is **terminal**: the loop stops and serves that error, no failover — reproducing "this error class doesn't fail over." It composes with the request-level `provider.allow_fallbacks:false` (either signal makes an error terminal). A dev models an error *class* by setting `fallthrough:false` on that class's fixture (e.g. the 403) and leaving 429/503 fixtures to default — so they can reproduce the exact failure that burned them and assert their handling, *and* the positive "fallback saved me" path. ### Correctness - **Load-time validation**: `validateFixtures` rejects a non-boolean `fallthrough` (`"false"`, `0`, `null`) — without this, a JSON author's `"false"` string would silently invert TERMINAL→fall-through (fail-closed quietly failing open). - A winning non-error slug is echoed as `response.model`; a terminal error is served as the OpenRouter error envelope (`{error:{code,message,metadata?}}`, no `model` field). - Default path is byte-identical to #328's fall-through. ### Testing / review - Red→green against the real HTTP surface; new tests for `fallthrough:false` terminal, default fall-through, composition, and the non-boolean load-time rejection. - Full suite green (`pnpm test`); tsc/lint/build/exports clean. No version bump (release is separate). - Reviewed via a multi-round CR (converged; silent-failure clean, gate/threading/validation confirmed correct). 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_018yy1y2Cwc31DaB4UNqvT8K
jpr5
added a commit
that referenced
this pull request
Jul 23, 2026
…tion) (#333) ## OpenRouter chat drift leg (cost-safe live canary + static envelope triangulation) Follow-up to #328, which shipped the OpenRouter chat/router mock (`src/openrouter-chat.ts`, `/api/v1/chat/completions` etc.) with **zero drift coverage**. This adds `src/__tests__/drift/openrouter-chat.drift.ts`, mirroring the existing `openrouter-video.drift.ts` pattern **cost-safely** (no paid completions). ### Two parts 1. **Live canary (FREE, key-gated)** — a new `listOpenRouterModels()` GETs the public `GET /api/v1/models` catalog (metadata only, never a paid generation) and asserts the chat-model families aimock mirrors are still present and the model-object schema hasn't drifted. `describe.skipIf(!OPENROUTER_API_KEY)` — runs in CI where the secret exists, skips cleanly locally. 2. **Static envelope triangulation (no key)** — drives the real aimock handler over HTTP and triangulates the OpenRouter chat envelopes it emits (non-streaming `gen-` id / top-level `provider` / `native_finish_reason` / `usage`+`cost_details` / `system_fingerprint` / `service_tier`, a streaming chunk, and the error envelope) against hand-authored conformant exemplars. Registers the `openrouter-chat` surface in `surface-registry.ts`. `OPENROUTER_API_KEY` is already a repo secret and wired into `test-drift.yml`. ### Review - Drift-detection demonstrated: perturbing an exemplar produced a real `API DRIFT DETECTED / TYPE MISMATCH` report; reverted → green. - CR caught and fixed a **`system_fingerprint` false-pass** (existence-only assertion) — now value-asserted and **mutation-proven** (forcing the mock to emit `null` fails the leg); plus a named `json.data` catalog-drift guard and an empty-catalog guard so the canary fails loudly, never silently. - Tests-inclusive typecheck clean (the drift file is under `src/__tests__`, excluded from the base `tsc`); leg green (static pass / live skip); build green. No version bump. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_018yy1y2Cwc31DaB4UNqvT8K
jpr5
added a commit
that referenced
this pull request
Jul 23, 2026
…safety) (#331) ## Why `scripts/update-competitive-matrix.ts` runs weekly (`.github/workflows/update-competitive-matrix.yml`) and **auto-commits to public, customer-facing docs** (`docs/index.html` and `docs/migrate-from-*/index.html`). Six pre-existing bugs (all predate #328) caused it to write wrong data or silently skip updates. This PR fixes all six and adds real red→green regression coverage. `#328`'s single new FEATURE_RULES entry ("OpenRouter router / fallback simulation") and its migration-variant entry are **left untouched** and are covered by an explicit regression guard. ## The 6 bugs & fixes | # | Bug | Fix | |---|-----|-----| | 1 | `COMPETITOR_MIGRATION_PAGES` mapped to `docs/migrate-from-*.html`, but the pages are **directories** (`.../index.html`), so `existsSync` always failed → every migration-page update silently skipped. `mokksy/ai-mocks` had no mapping at all. | Paths → `/index.html`; added the `mokksy/ai-mocks` mapping. (Note: `VidaiMock` was already mapped; the genuinely missing competitor was `mokksy/ai-mocks`.) | | 2 | Unanchored keyword regexes: `cli`→"client"/"click", `sse`→"assess" → false ✗→✓ flips in public docs. | New `keywordRegex()` wraps each keyword in non-alphanumeric boundary lookarounds `(?<![a-z0-9])…(?![a-z0-9])`; works for literal tokens, path/regex keywords, and edge-punctuation keywords alike. | | 3 | `countProviders` over-counted via substrings (`cohere`→"coherent", `aws`→"flaws") plus a redundant `gemini.*interactions` group double-counting Gemini. | Bounded matching (same helper) + removed the redundant group (13 groups, each provider counted once). | | 4 | `String.replace(pattern, htmlString)` let literal `$`, `$&`, `$1` in HTML be interpreted → corrupted/duplicated output (3 sites). | Switched the HTML-derived replacements to **function form** `(…) => text` so replacement text stays literal. | | 5 | Migration feature-cell updater assumed the competitor column was adjacent to the label; the provider-count updater located it by header **name** — they disagreed when an intervening column (e.g. aimock) existed. | Both now use one shared `findMigrationCompetitorColumn()` (token-aware header-name match — avoids the `"vidaimock"⊇"aimock"` substring trap). | | 6 | Orphaned variant key `"Realtime translate/whisper"` never matched its rule label `"Realtime transcription/translation"` → that rule got no migration variants, cells never flipped. | Renamed the key to match the rule label exactly. | ## Tests A 1168-line `competitive-matrix.test.ts` already existed but **reimplemented the script logic inline** (a mirror), so it tested copies that mirrored the bugs — it could never catch them. Converted it to **import the real script** (added `export`s + an `import.meta` main-guard so importing doesn't run `main()`), fixed the one assertion that encoded the buggy behavior (`countProviders` all-groups `14`→`13`), and added red→green repros for each bug plus a regression guard for the OpenRouter entry. ## Red → Green (fixture-based, no network) **RED** — bug repros run against the pre-fix script (enabling refactor only, no behavioral change): ``` Tests 13 failed | 49 passed (62) ``` The 13 failures were exactly: - Bug 1: `maps every mapped competitor to a file that actually exists on disk`; `includes a mapping for the mokksy/ai-mocks competitor` - Bug 2: `does not flip "CLI server" from the words "client"/"click"`; `does not flip "Chat Completions SSE" from the word "assess"` - Bug 3: `does not count "cohere" inside "coherent" or "aws" inside "flaws"`; `counts Gemini exactly once`; `counts all 13 provider groups …` - Bug 4: `applyChanges does not duplicate the row when replacement text contains $&`; `updateProviderCounts does not corrupt the table when a cell contains $&` - Bug 5: `flips the competitor cell even when aimock is the first data column`; `resolves the column when the header text differs from the competitor key (Mokksy)` - Bug 6: `returns variants for the real rule label 'Realtime transcription/translation'`; `flips a migration row that uses a variant label for the realtime rule` (The #328 OpenRouter guard tests and the positive-control tests passed in the RED run, as expected.) **GREEN** — after the 6 fixes: ``` ✓ src/__tests__/competitive-matrix.test.ts (62 tests) Test Files 1 passed (1) Tests 62 passed (62) ``` **Full suite / build / lint** (no regressions): ``` Tests 4706 passed | 44 skipped (4750) # vitest run Build complete # pnpm build (tsdown) All matched files use Prettier code style! # pnpm format:check eslint . → 0 problems # pnpm lint ``` **Live end-to-end** — `npx tsx scripts/update-competitive-matrix.ts --dry-run` against real GitHub still fires `main()`, parses all 4 competitors + the 38-row / 6-column matrix, and reports "No changes detected" (no spurious flips). ## Scope / safety - No package version bump (this is scripts/docs tooling, not the published package surface). - `#328`'s FEATURE_RULES + variant entry unchanged; guarded by regression tests. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KamK73Wu5heLxJEogM6hHC
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
OpenRouter chat/LLM router simulation
Adds aimock's first OpenRouter chat/router integration. Where aimock already mocks OpenRouter's video surface, this adds the chat/LLM surface — and, more importantly, deterministically simulates OpenRouter's routing layer offline (model fallback / failover, provider selection, cost), a category no existing mock fills.
What it does
/api/v1/base prefix (captured before the compat-normalizer rewrites it). OpenAI/v1/…callers are byte-for-byte unchanged (pinned by tests, incl. streaming).gen-id, top-levelprovider(winning slug's author by default, fixture-overridable), bothfinish_reasonandnative_finish_reason, always-presentsystem_fingerprint/service_tier, and rich fixture-scriptableusage—cost, three-fieldcost_details,is_byok, and token-detail objects.models[]fallback simulation — the wedge: iterate[model, ...models], serve the first non-error match, fall through 429/503 runtime-error candidates, honorprovider.allow_fallbacks:false(fail-closed), and echo the winning slug asresponse.model. Lets a test assert "primary is down → my app consumed the fallback" with zero network.GET /api/v1/models,/api/v1/key,/api/v1/credits.{ error: { code, message, metadata? } }withcode == HTTP status.: OPENROUTER PROCESSINGSSE keepalive.Wire fidelity — verified against real bytes
Field shapes were adversarially verified against three independent sources before implementation: OpenRouter's OpenAPI spec, both official SDKs' source (
@openrouter/sdk,@openrouter/ai-sdk-provider), and a live capture with a real key. This caught that the spec'schatcmpl-id example and nested-provider shape do not match real runtime bytes (which usegen-ids and a top-levelprovider) — a deviation that docs-only research would have shipped.Competitive matrix + drift automation
Adds an "OpenRouter router / fallback simulation" row to the landing competitive matrix (aimock ✓; MSW / VidaiMock / mock-llm / piyook / mokksy ✗ — none mock OpenRouter's router), plus a
FEATURE_RULESentry so the weekly drift job tracks whether any competitor ships it.Testing / review
pnpm test); 20+ new tests including OpenAI-streaming byte-isolation, tool-call shaping,allow_fallbacks:falsefail-closed, and scriptable-cost.Follow-ups (intentionally out of scope)
reasoning_detailsmirror, and the deferred discovery/legacy endpoints.scripts/update-competitive-matrix.ts(broken migration-page paths, unanchored keyword regexes, loose provider counting) — predate this change; our added tracking row is unaffected (verified).🤖 Generated with Claude Code
https://claude.ai/code/session_018yy1y2Cwc31DaB4UNqvT8K