Replayable Cost Fixtures for OpenAI-Compatible Agent SDKs (August 2026)

[chawkian]

One SDK feature I would like to see become normal is a replayable cost fixture: a small, committed test artifact that lets a team prove that routing, token accounting, cache handling, and fallback policy still produce the expected cost envelope after an SDK or gateway change.

This is related to budget estimation, but it is not the same thing. A pre-dispatch estimate protects the live request. A replay fixture protects the next release. If an application routes across DeepSeek, Kimi, GLM, Qwen, and ERNIE through one OpenAI-compatible request shape, a small SDK change can alter the selected model, output cap, cache assumption, retry behavior, or billing group inheritance. Unit tests should catch that before production traffic does.

For today's check, August 21, 2026, I pulled the live AIWave /api/pricing feed before writing. The snapshot returned pricing_version=a42d372ccf0b5dd13ecf71203521f9d2, with group_ratio showing default=3 and vip=1. A few rows from that live feed:

• deepseek-v4-flash: $0.319/M input, $0.957/M output, $0.0318/M cached input.
• deepseek-v4-pro: $0.957/M input, $2.871/M output, $0.0333/M cached input.
• kimi-k2.6: $0.545/M input, about $2.300/M output, about $0.1714/M cached input.
• kimi-k3: $2.250/M input, $11.250/M output, $0.200/M cached input.
• glm-5.1: $1.050/M input, about $3.300/M output, $0.32381/M cached input.
• ernie-5.0: $1.232877/M input, about $4.657/M output, $0.100/M cached input.
• qwen3.5-122b-a10b: $0.246575/M input and about $1.862/M output.

The exact values matter less than the discipline around them. A fixture should record the source, checked time, pricing version if available, selected model, expected token shape, cache expectation, output cap, fallback permission, and billing group behavior. Then CI can replay the route decision without calling the model.

I would start with a fixture shape like this:

{
  "name": "support-summary-medium-context",
  "pricing_source": "https://aiwave.live/api/pricing",
  "pricing_checked_at": "2026-08-21T14:00:00Z",
  "pricing_version": "a42d372ccf0b5dd13ecf71203521f9d2",
  "billing_group": "vip",
  "request": {
    "task_kind": "support_summary",
    "estimated_input_tokens": 18000,
    "estimated_cached_input_tokens": 12000,
    "max_output_tokens": 900
  },
  "expected": {
    "model": "deepseek-v4-flash",
    "fallback_allowed": true,
    "max_estimated_usd": 0.005,
    "route_reason": "bounded_summary"
  }
}

The test does not need to assert every decimal forever. In fact, overly strict decimal tests can create noise when rate cards are intentionally refreshed. I would assert behavior and tolerances: the selected model family, whether cache is counted separately, whether the route stays under an agreed budget band, whether fallback remains allowed, and whether VIP status inherits correctly instead of requiring a separate token setting.

This also helps with incident response. If a customer reports that a workflow became more expensive after a deploy, the team can replay the old fixture against the new SDK and ask specific questions. Did the cache field disappear? Did output caps increase? Did a retry path start using Pro instead of Flash? Did a default-group fixture accidentally replace a VIP fixture? Those are concrete diffs, not guesses.

There are two traps I would avoid.

First, do not use live price fetching inside every CI run as the only test. That makes tests unstable and hides whether the SDK changed. Keep a committed fixture for regression tests, then run a separate freshness job that tells you when the fixture should be updated.

Second, do not make the fixture a finance-only artifact. It should live close enough to the SDK examples that developers see it when they add a route, a fallback, or a cache feature.

Discussion question: should replayable cost fixtures be part of the Python SDK examples, generated by the gateway, or both? My bias is both: the gateway knows the current rate card, but SDK fixtures make cost behavior reviewable in pull requests. If you already test this, what fields have been most useful, and what tolerances keep the tests stable?