Error-class-bound fallthrough for OpenRouter models[] fallback (#334)

## Error-class-bound fallthrough for the OpenRouter `models[]` fallback
engine

Follow-up to #328. That PR shipped `models[]` failover (an error-fixture
candidate falls through to the next model, unless
`provider.allow_fallbacks:false`). But real OpenRouter fails over
**inconsistently by error class** — the loudest authentic user complaint
(openclaw#60191, #45834, #19249): a `403 budget-exceeded` or generic
"provider error" does **not** advance to the next model (it retries the
same primary), while 429/503 do. Users can't rehearse that.

### What this adds
An optional **`fallthrough?: boolean`** on the error fixture response:
- **absent (default)** → current behavior: the error candidate falls
through to the next `models[]` candidate (fully backward-compatible).
- **`fallthrough: false`** → the error is **terminal**: the loop stops
and serves that error, no failover — reproducing "this error class
doesn't fail over."

It composes with the request-level `provider.allow_fallbacks:false`
(either signal makes an error terminal). A dev models an error *class*
by setting `fallthrough:false` on that class's fixture (e.g. the 403)
and leaving 429/503 fixtures to default — so they can reproduce the
exact failure that burned them and assert their handling, *and* the
positive "fallback saved me" path.

### Correctness
- **Load-time validation**: `validateFixtures` rejects a non-boolean
`fallthrough` (`"false"`, `0`, `null`) — without this, a JSON author's
`"false"` string would silently invert TERMINAL→fall-through
(fail-closed quietly failing open).
- A winning non-error slug is echoed as `response.model`; a terminal
error is served as the OpenRouter error envelope
(`{error:{code,message,metadata?}}`, no `model` field).
- Default path is byte-identical to #328's fall-through.

### Testing / review
- Red→green against the real HTTP surface; new tests for
`fallthrough:false` terminal, default fall-through, composition, and the
non-boolean load-time rejection.
- Full suite green (`pnpm test`); tsc/lint/build/exports clean. No
version bump (release is separate).
- Reviewed via a multi-round CR (converged; silent-failure clean,
gate/threading/validation confirmed correct).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_018yy1y2Cwc31DaB4UNqvT8K

Author: jpr5

docs/openrouter-chat/index.html

@@ -344,6 +344,37 @@ <h2 id="fallback">models[] Fallback Simulation</h2>
           primary candidate is tried, and its error (or a miss) is terminal instead of walking the
           rest of the list.
         </p>
+        <p>
+          Real OpenRouter fails over <em>inconsistently</em> by error class: a
+          <code>403 budget-exceeded</code> or a generic "provider returned error" is served as
+          terminal (it does <em>not</em> advance to the next candidate), while 429/503 usually do
+          fail over. Reproduce that per error class with <code>fallthrough: false</code> on the
+          error fixture &mdash; the loop stops and serves that error as terminal, no failover, even
+          when a good candidate follows. Absent (or <code>true</code>) keeps the default
+          fall-through, so existing fixtures are unchanged. This composes with the request-level
+          gate: fall-through happens only when both allow it (if either
+          <code>allow_fallbacks: false</code> or <code>fallthrough: false</code> says stop, the
+          error is terminal). Model an error class a dev must handle themselves &mdash; the exact
+          failure that burned them &mdash; by setting <code>fallthrough: false</code> on that
+          class's fixture (e.g. the 403) and leaving 429/503 fixtures to default.
+        </p>
+        <div class="code-block">
+          <div class="code-block-header">
+            terminal-error.test.ts <span class="lang-tag">ts</span>
+          </div>
+          <pre><code><span class="cm">// a 403 that does NOT fail over — the app must handle it, no fallback</span>
+<span class="op">mock</span>.<span class="fn">on</span>(
+  { <span class="prop">model</span>: <span class="str">"openai/gpt-4o"</span>, <span class="prop">userMessage</span>: <span class="str">"route"</span> },
+  { <span class="prop">error</span>: { <span class="prop">message</span>: <span class="str">"budget exceeded"</span> }, <span class="prop">status</span>: <span class="num">403</span>, <span class="prop">fallthrough</span>: <span class="kw">false</span> }
+);
+<span class="op">mock</span>.<span class="fn">on</span>(
+  { <span class="prop">model</span>: <span class="str">"anthropic/claude-3.5-sonnet"</span>, <span class="prop">userMessage</span>: <span class="str">"route"</span> },
+  { <span class="prop">content</span>: <span class="str">"never reached"</span> }
+);
+
+<span class="cm">// models[] lists a good fallback, but the 403 is terminal — no failover</span>
+<span class="fn">expect</span>(<span class="op">res</span>.<span class="prop">status</span>).<span class="fn">toBe</span>(<span class="num">403</span>);</code></pre>
+        </div>
         <div class="info-box">
           <p>
             <strong>Deliberate non-goal.</strong> Real OpenRouter rejects an unknown/invalid model

skills/write-fixtures/SKILL.md

@@ -485,6 +485,7 @@ A request whose path starts with `/api/v1/` (point the OpenAI SDK at a `baseURL`
 
 - **Scriptable cost / provider / finish reason**: set `provider`, `nativeFinishReason`, and `usage.cost` on the response (see the overrides table). `cost`/`cost_details` are emitted only when a fixture supplies `cost` — aimock never fabricates a cost.
 - **`models[]` fallback (router failover)**: when the request body carries `models: [m1, m2, ...]`, aimock walks `[model, ...models]` in order and serves the first fixture that returns a NON-error response. A `429`/`503` error fixture on a candidate simulates a RUNTIME provider failure and falls through to the next candidate; the winning slug is echoed back as the top-level `model` (assert failover via `response.model`). Model the primary's "failure" as a 429/503 — an unknown/invalid model is just a fixture miss (aimock does not replicate OpenRouter's up-front invalid-model 400).
+- **Terminal (non-failover) error class — `fallthrough: false`**: real OpenRouter fails over inconsistently by error class (a `403 budget-exceeded` / generic "provider returned error" is served as terminal and does NOT advance to the next candidate, while 429/503 usually do). Set `fallthrough: false` on an error fixture to make it terminal: the fallback loop stops and serves that error even when a good candidate follows. Absent / `true` keeps the default fall-through. Composes with `provider.allow_fallbacks`: fall-through happens only when both allow it (if either says stop, the error is terminal). Use it to reproduce the exact provider error a dev's app must handle itself. `{ error: { message: "budget exceeded" }, status: 403, fallthrough: false }`
 - **Keepalive**: set the fixture option `openRouterProcessing: true` to emit one `: OPENROUTER PROCESSING` SSE comment before the first data frame (opt-in, default off).
 - **Error envelope**: OpenRouter errors are `{ error: { message, code } }` (numeric `code` == HTTP status), with optional free-form `metadata`.
 

src/__tests__/openrouter-chat.test.ts

@@ -605,6 +605,183 @@ describe("OpenRouter chat: fail-closed + fallback correctness (round-1 fixes)",
   });
 });
 
+// ---------------------------------------------------------------------------
+// v1.1 — error-class-bound fallthrough (`fallthrough: false` = terminal error)
+// ---------------------------------------------------------------------------
+
+describe("OpenRouter chat: error-class-bound fallthrough (v1.1)", () => {
+  let tmpDir: string | null = null;
+  afterEach(() => {
+    if (tmpDir) {
+      rmSync(tmpDir, { recursive: true, force: true });
+      tmpDir = null;
+    }
+  });
+
+  // A 403 whose error CLASS does not fail over on real OpenRouter (the
+  // openclaw #60191 pattern): the primary error is marked `fallthrough: false`,
+  // so it must be served as terminal even though a good fallback follows.
+  const terminalChain = (fallthrough?: boolean): Fixture[] => [
+    {
+      match: { model: "primary/bad", userMessage: "route" },
+      response: {
+        error: { message: "budget exceeded (non-failover class)" },
+        status: 403,
+        ...(fallthrough !== undefined && { fallthrough }),
+      },
+    },
+    {
+      match: { model: "fallback/good", userMessage: "route" },
+      response: { content: "served by fallback", usage: { cost: 0.5 } },
+    },
+  ];
+
+  it("fallthrough:false serves the primary error as terminal — NO failover", async () => {
+    instance = await createServer(terminalChain(false));
+    const res = await httpPost(`${instance.url}${OR}`, {
+      model: "primary/bad",
+      models: ["primary/bad", "fallback/good"],
+      messages: [{ role: "user", content: "route" }],
+    });
+    const json = JSON.parse(res.body);
+    // Terminal: the primary's 403 is served, the good fallback is never reached.
+    expect(res.status).toBe(403);
+    expect(json.error.code).toBe(403);
+    expect(json.error.message).toBe("budget exceeded (non-failover class)");
+  });
+
+  it("fallthrough absent (default) still falls through to the good fallback", async () => {
+    instance = await createServer(terminalChain(undefined));
+    const res = await httpPost(`${instance.url}${OR}`, {
+      model: "primary/bad",
+      models: ["primary/bad", "fallback/good"],
+      messages: [{ role: "user", content: "route" }],
+    });
+    const json = JSON.parse(res.body);
+    expect(res.status).toBe(200);
+    expect(json.model).toBe("fallback/good");
+    expect(json.choices[0].message.content).toBe("served by fallback");
+  });
+
+  it("fallthrough:true (explicit) still falls through to the good fallback", async () => {
+    instance = await createServer(terminalChain(true));
+    const res = await httpPost(`${instance.url}${OR}`, {
+      model: "primary/bad",
+      models: ["primary/bad", "fallback/good"],
+      messages: [{ role: "user", content: "route" }],
+    });
+    const json = JSON.parse(res.body);
+    expect(res.status).toBe(200);
+    expect(json.model).toBe("fallback/good");
+  });
+
+  it("fallthrough:false composes with allow_fallbacks:true — still terminal", async () => {
+    // The per-error gate is independent of the request-level gate: if EITHER
+    // says don't fall through, don't. allow_fallbacks:true does not override
+    // fallthrough:false.
+    instance = await createServer(terminalChain(false));
+    const res = await httpPost(`${instance.url}${OR}`, {
+      model: "primary/bad",
+      models: ["primary/bad", "fallback/good"],
+      provider: { allow_fallbacks: true },
+      messages: [{ role: "user", content: "route" }],
+    });
+    expect(res.status).toBe(403);
+    expect(JSON.parse(res.body).error.message).toBe("budget exceeded (non-failover class)");
+  });
+
+  it("fallthrough:false on an error does not affect a normal (non-error) fallback chain", async () => {
+    // Shipped behavior intact: a SUCCESS primary is served regardless of any
+    // fallthrough flag on a sibling error fixture; the terminal-error flag only
+    // gates the error candidate it lives on.
+    const fixtures: Fixture[] = [
+      {
+        match: { model: "primary/good", userMessage: "route" },
+        response: { content: "served by primary" },
+      },
+      {
+        match: { model: "other/bad", userMessage: "route" },
+        response: { error: { message: "non-failover" }, status: 403, fallthrough: false },
+      },
+    ];
+    instance = await createServer(fixtures);
+    const res = await httpPost(`${instance.url}${OR}`, {
+      model: "primary/good",
+      models: ["primary/good", "other/bad"],
+      messages: [{ role: "user", content: "route" }],
+    });
+    const json = JSON.parse(res.body);
+    expect(res.status).toBe(200);
+    expect(json.model).toBe("primary/good");
+    expect(json.choices[0].message.content).toBe("served by primary");
+  });
+
+  it("threads fallthrough:false through the real JSON fixture-loader path", async () => {
+    tmpDir = mkdtempSync(join(tmpdir(), "or-fallthrough-fixture-"));
+    const fixtureFile = join(tmpDir, "fallthrough.json");
+    writeFileSync(
+      fixtureFile,
+      JSON.stringify({
+        fixtures: [
+          {
+            match: { model: "primary/bad", userMessage: "route" },
+            response: {
+              error: { message: "budget exceeded (non-failover class)" },
+              status: 403,
+              fallthrough: false,
+            },
+          },
+          {
+            match: { model: "fallback/good", userMessage: "route" },
+            response: { content: "served by fallback" },
+          },
+        ],
+      }),
+      "utf-8",
+    );
+    const loaded = loadFixtureFile(fixtureFile);
+    const errors = validateFixtures(loaded).filter((r) => r.severity === "error");
+    expect(errors).toEqual([]);
+
+    instance = await createServer(loaded);
+    const res = await httpPost(`${instance.url}${OR}`, {
+      model: "primary/bad",
+      models: ["primary/bad", "fallback/good"],
+      messages: [{ role: "user", content: "route" }],
+    });
+    expect(res.status).toBe(403);
+    expect(JSON.parse(res.body).error.message).toBe("budget exceeded (non-failover class)");
+  });
+
+  it("rejects a non-boolean fallthrough at load (fail-closed cannot fail open)", async () => {
+    // A JSON/YAML author writing "false" (string), 0, or null would otherwise
+    // be silently treated as fall-through (`!== false` is true), the OPPOSITE
+    // of the intended terminal behavior. The loader must reject it up front.
+    const bad: Fixture[] = [
+      {
+        match: { model: "primary/bad", userMessage: "route" },
+        // Intentionally wrong type — mimics a JSON author's "false" string.
+        response: {
+          error: { message: "x" },
+          status: 403,
+          fallthrough: "false",
+        } as unknown as Fixture["response"],
+      },
+    ];
+    const errors = validateFixtures(bad).filter((r) => r.severity === "error");
+    expect(errors.some((e) => /fallthrough must be a boolean/.test(e.message))).toBe(true);
+
+    // A proper boolean is accepted (no error).
+    const good: Fixture[] = [
+      {
+        match: { model: "primary/bad", userMessage: "route" },
+        response: { error: { message: "x" }, status: 403, fallthrough: false },
+      },
+    ];
+    expect(validateFixtures(good).filter((r) => r.severity === "error")).toEqual([]);
+  });
+});
+
 describe("OpenRouter request extensions", () => {
   it("accepts and journals extension fields + attribution headers without rejecting", async () => {
     const fixtures: Fixture[] = [{ match: { userMessage: "hi" }, response: { content: "ok" } }];

src/fixture-loader.ts

@@ -52,7 +52,13 @@ import {
   handleOpenRouterKey,
   handleOpenRouterCredits,
 } from "./openrouter-chat.js";
-import type { FixtureResponse, ChatCompletion, SSEChunk, ResponseOverrides } from "./types.js";
+import type {
+  FixtureResponse,
+  ErrorResponse,
+  ChatCompletion,
+  SSEChunk,
+  ResponseOverrides,
+} from "./types.js";
 import { handleResponses } from "./responses.js";
 import { handleMessages } from "./messages.js";
 import { handleGemini } from "./gemini.js";
@@ -701,13 +707,23 @@ async function handleCompletions(
       // request see the increment (see the `let matchCounts` note above).
       matchCounts = journal.getFixtureMatchCountsForTest(testId);
       responseModel = candidate;
-      if (outcome.isError && allowFallbacks) {
-        // Runtime provider failure — remember it and fail over to the next.
-        lastErrorFixture = outcome.fixture;
-        lastErrorResponse = outcome.response;
-        continue;
+      if (outcome.isError) {
+        // Per-error-fixture failover gate. An ERROR fixture may set
+        // `fallthrough: false` to mark its error CLASS as non-failover-eligible
+        // — reproducing OpenRouter serving a 403/generic provider error as
+        // terminal instead of advancing to the next `models[]` candidate
+        // (openclaw #60191). Absent / `true` keeps the default fall-through.
+        // Composes with the request-level `allowFallbacks` gate: fail over only
+        // when BOTH allow it (if EITHER says don't, this candidate is terminal).
+        const errorFallsThrough = (outcome.response as ErrorResponse).fallthrough !== false;
+        if (allowFallbacks && errorFallsThrough) {
+          // Runtime provider failure — remember it and fail over to the next.
+          lastErrorFixture = outcome.fixture;
+          lastErrorResponse = outcome.response;
+          continue;
+        }
       }
-      // Success, or a primary error under allow_fallbacks:false — terminal.
+      // Success, or a terminal error (allow_fallbacks:false or fallthrough:false).
       fixture = outcome.fixture;
       preResolvedResponse = outcome.response;
       break;

src/server.ts

@@ -46,10 +46,17 @@ export interface ChatCompletionRequest {
   /**
    * OpenRouter fallback model list. When present (and the request arrived on
    * the OpenRouter `/api/v1/...` base), the candidate list `[model, ...models]`
-   * is attempted in order and the first NON-error fixture match wins — the
-   * winning slug is echoed back as the response `model`. Purely additive: a
-   * request without `models` behaves as a single-model match. Every other
-   * OpenRouter request extension (`provider`, `route`, `reasoning`, `plugins`,
+   * is attempted in order. A NON-error fixture match wins; an ERROR fixture
+   * candidate falls through to the next candidate by default — UNLESS it is
+   * terminal, in which case that error is served with no failover. An error is
+   * terminal when its fixture sets `fallthrough: false` (per-error-class gate)
+   * or when the request sets `provider.allow_fallbacks: false` (request-level
+   * gate). A winning (non-error) slug is echoed back as the response `model`;
+   * when the chain terminates on an error, the response is the OpenRouter error
+   * envelope (`{ error: { code, message, metadata? } }`), which carries no
+   * `model` field. Purely additive: a request without `models` behaves as a
+   * single-model match. Every other
+   * OpenRouter request extension (`route`, `reasoning`, `plugins`,
    * `prediction`, `usage`, and any unknown key) is accepted and journaled via
    * the index signature below — never required, never rejected. aimock does not
    * model those sub-shapes; unknown body keys pass straight through.
@@ -347,6 +354,22 @@ export interface ErrorResponse {
   status?: number;
   /** Override the Retry-After header value on 429 responses. Default: 1. */
   retryAfter?: number;
+  /**
+   * OpenRouter `models[]` failover eligibility for THIS error class. Real
+   * OpenRouter fails over INCONSISTENTLY by error class — a `403 budget-exceeded`
+   * or a generic "provider returned error" is served as terminal (it does NOT
+   * advance to the next `models[]` candidate), while 429/503 usually do fail
+   * over (openclaw #60191). This flag lets a fixture reproduce that:
+   * - absent / `true` (default) — the error candidate FALLS THROUGH to the next
+   *   `models[]` candidate (backward-compatible; existing fixtures unchanged).
+   * - `false` — the error is TERMINAL: the fallback loop stops and serves this
+   *   error with no failover, even if a later candidate would match.
+   *
+   * Composes with the request-level `provider.allow_fallbacks`: fall-through
+   * happens only when BOTH allow it (if EITHER says don't, don't). Only
+   * consulted for OpenRouter `models[]` fallback; ignored elsewhere.
+   */
+  fallthrough?: boolean;
 }
 
 export interface EmbeddingResponse {