Evaluation Baseline Manifests for OpenAI-Compatible Agent SDKs (August 2026)

[chawkian]

One SDK feature I would like to see discussed more seriously is an evaluation baseline manifest. Many teams already run nightly model evaluations, but the inputs that define those runs are often scattered across test files, CI variables, pricing spreadsheets, and dashboards. That makes a failing evaluation hard to interpret. Did the model regress, did the route change, did the price card change, or did the test harness simply use a different fixture set?

For an OpenAI-compatible SDK, I would treat an evaluation run as a reproducible contract. The SDK should be able to emit a manifest before the run starts, and that manifest should be saved beside the results. The goal is not to make every evaluation deterministic. The goal is to make every meaningful input visible enough that another engineer can explain the result one week later.

For today's check, August 26, 2026, I pulled the live AIWave /api/pricing feed before drafting. It returned success=true, pricing_version=a42d372ccf0b5dd13ecf71203521f9d2, and group_ratio values of default=3 and vip=1. Example VIP-baseline rows from that feed:

• deepseek-v4-flash: $0.319/M input, $0.957/M output, about $0.0101/M cached input.
• deepseek-v4-pro: $0.957/M input, $2.871/M output, about $0.0319/M cached input.
• qwen3.5-122b-a10b: $0.246575/M input and about $1.863/M output.
• kimi-k3: $2.250/M input, $11.250/M output, $0.450/M cached input.

I also rechecked the public AIWave pricing and predictable-pricing pages, DeepSeek pricing docs, QwenCloud pricing docs, Kimi pricing docs, and Z.AI GLM docs during the run; all returned HTTP 200. That matters for evaluation because a "better" model score can be misleading if the benchmark report does not preserve the rate card used to calculate cost per accepted answer.

The manifest shape I would start with is small and explicit:

{
  "evaluation_manifest": {
    "sdk": "aiwave-sdk-python",
    "created_at": "2026-08-26",
    "route_policy": "quality_then_budget",
    "pricing_source": "https://aiwave.live/api/pricing",
    "pricing_version": "a42d372ccf0b5dd13ecf71203521f9d2",
    "fixtures_sha256": "recorded-by-ci",
    "models": [
      "deepseek-v4-flash",
      "deepseek-v4-pro",
      "qwen3.5-122b-a10b",
      "kimi-k3"
    ],
    "limits": {
      "max_input_tokens": 32000,
      "max_output_tokens": 4096,
      "max_tool_calls": 4,
      "max_eval_usd": 25
    }
  }
}

The important field is fixtures_sha256. Without a stable fixture digest, a nightly run can drift while still looking like the same benchmark. A product team may add longer support tickets, a code team may add harder stack traces, or a retrieval team may add documents with very different token profiles. Those are valid changes, but they should be visible changes.

The second important field is pricing_version. If the evaluation dashboard tracks accuracy, latency, and cost, the cost side should not be recomputed later against a different rate card without a label. Teams need both numbers: the original cost estimate from the run, and any later restatement under the current price card. That distinction helps avoid false conclusions when provider rates, cache ratios, or output multipliers change.

I would also log route decisions at the case level. A single aggregate score can hide whether the SDK chose the same model for every fixture, or whether the route policy moved code-heavy prompts to one model and long-context prompts to another. A useful row would include fixture_id, model, route_reason, input_tokens, cached_input_tokens, output_tokens, estimated_usd, latency_ms, and accepted.

My bias is that the SDK should create the manifest, while CI should store and compare it. The gateway can enforce the freshest account limits, but the SDK is closer to the fixtures, tool settings, and route policy that make the evaluation reproducible.

Discussion question: should evaluation manifests be a first-class SDK object, or is this better left to CI templates and external observability tools? I lean toward first-class support because the SDK can capture the request and routing context before it is flattened into logs.