Variance Evidence Exports for OpenAI-Compatible SDK Runs (September 2026)

[chawkian]

After an AI workload finishes, the question is often not "did the SDK return a response?" The harder question is "can the application explain why this run cost more or less than expected?"

For small demos, a returned message and a dashboard row may be enough. For production agent runs, batch jobs, and support workflows, that gap becomes expensive. The application knows the feature, job id, rollout window, retry policy, and expected workload shape. The gateway knows the resolved route, account group, pricing version, token buckets, cache treatment, and final request status. If those facts stay separated, spend variance turns into a manual investigation.

I checked AIWave's live pricing endpoint for this discussion on Sep 3, 2026. The /api/pricing response returned success=true, 63 route records, pricing version a42d372ccf0b5dd13ecf71203521f9d2, auto_groups=["default"], and group ratios of default=3 and vip=1. A few route fields from that same snapshot were:

Model	model_ratio	completion_ratio	cache_ratio	Enabled groups
deepseek-v4-flash	0.319	3	0.0318	default, vip, svip
deepseek-v4-pro	0.957	3	0.0333	default, vip, svip
glm-5.1	1.05	3.142857	0.32381	default, vip, svip
kimi-k3	2.25	5	0.2	default, vip, svip
qwen3-max	0.7810988945590882	4	absent	default, vip, svip

Those fields are gateway catalog inputs, not a complete invoice. The actual debit still depends on the account group, token counts, cache-hit bucket, output size, retries, fallback behavior, and the pricing version accepted at request time. That is why I think SDKs should make variance evidence export a first-class opt-in feature.

The export does not need raw prompts, completions, customer emails, API keys, or payment identifiers. It should be a small structured record that the application can write beside its own job log:

{
  "run_id": "agent_run_2026_09_03_042",
  "feature": "support_summary",
  "requested_model": "deepseek-v4-flash",
  "served_model": "deepseek-v4-flash",
  "pricing_version": "a42d372ccf0b5dd13ecf71203521f9d2",
  "effective_group": "vip",
  "input_tokens": 2200000,
  "cached_input_tokens": 1600000,
  "output_tokens": 340000,
  "retry_count": 1,
  "fallback_from": null,
  "status": "success"
}

With records like this, a variance review can split the delta into useful buckets. Volume variance asks whether the feature sent more requests or larger prompts than planned. Mix variance asks whether the resolved model changed. Shape variance asks whether output, cache-hit share, or tool results moved. Policy variance asks whether retries, fallbacks, or account groups changed.

That split matters because the fix is different for each bucket. If output grew, the team may need a stricter max_tokens policy or a shorter answer contract. If cache share fell, the issue may be prompt-prefix stability. If the resolved model changed, the issue may be an alias, fallback rule, or route migration. If retries grew, the answer may belong in capacity policy rather than pricing code.

For aiwave-sdk-python, I would expose this through a callback or response-side extension rather than forcing every user to adopt a ledger schema:

def record_variance_evidence(evidence):
    app_ledger.write_jsonl(evidence.redacted())

client = AIWave(
    api_key="...",
    base_url="https://aiwave.live/v1",
    variance_evidence_hook=record_variance_evidence,
)

response = client.chat.completions.create(
    model="deepseek-v4-flash",
    messages=[{"role": "user", "content": "Summarize the support thread."}],
    max_tokens=900,
)

The important part is the boundary. The gateway should remain the source of truth for route identity, account group, and pricing version. The SDK should make those facts easy for the application to retain at the moment the response is handled. Later dashboards are still useful, but they should not be the only durable explanation.

I would also include a comparison key for planned versus observed usage. The application usually knows the budget assumption before the call starts:

{
  "expected": {"input_tokens": 1800000, "output_tokens": 180000},
  "observed": {"input_tokens": 2200000, "output_tokens": 340000}
}

That small addition turns the SDK from a passive client into a useful witness. It cannot decide whether the spend was acceptable. It can preserve the facts that make the decision possible.

Open question: should variance evidence live as an SDK callback, a response object extension, or a separate export API called after the request completes?