Streaming Usage Checkpoints for OpenAI-Compatible SDKs (August 2026)

[chawkian]

One SDK behavior I would like to see discussed more: streaming requests should expose usage checkpoints before the final completion event.

Most OpenAI-compatible SDKs make streaming feel simple. The application receives chunks, appends text to a buffer, and waits for the final event. That is a good developer experience for rendering. It is weaker for billing, retries, and support. The moment a long stream is interrupted, the application needs to answer several questions that the final response may never answer: how much output reached the client, how much output did the provider continue generating, which pricing snapshot was active, and whether a retry would double-spend the same task.

I checked live pricing before writing this on 2026-08-30. AIWave /api/pricing returned success=true, pricing version a42d372ccf0b5dd13ecf71203521f9d2, 63 model rows, auto_groups=["default"], and group ratios of default=3 and vip=1. The current DeepSeek rows exposed through AIWave include:

• deepseek-v4-flash: model_ratio=0.319, completion_ratio=3, cache_ratio=0.0318
• deepseek-v4-pro: model_ratio=0.957, completion_ratio=3, cache_ratio=0.0333
• glm-5.1: model_ratio=1.05, completion_ratio=3.142857, cache_ratio=0.32381
• kimi-k3: model_ratio=2.25, completion_ratio=5, cache_ratio=0.2

I also rechecked the official DeepSeek pricing page during this run. It still exposes deepseek-v4-flash and deepseek-v4-pro with 1M context, 384K maximum output, cache-hit input, cache-miss input, and output buckets split by off-peak and peak periods. The visible USD rows include Flash at $0.007/$0.014 cache-hit input, $0.22/$0.44 cache-miss input, and $0.66/$1.32 output per 1M tokens for off-peak/peak. Pro is higher: $0.022/$0.044, $0.66/$1.32, and $1.98/$3.96 on the same buckets.

Those numbers matter because streaming failures rarely happen at a convenient accounting boundary. Imagine a coding agent that streams a repository-scale answer. The browser receives 1,400 tokens, the user closes the tab, and the provider keeps producing for another second. The gateway later receives a final usage frame with 1,760 output tokens. If the app only sees "stream disconnected", it may retry the whole request without knowing whether the first run was mostly complete, partially billable, or already expensive enough to ask for approval.

I would like SDKs to make checkpoint events first-class. Not every chunk needs pricing math, but the SDK could periodically emit a small structured record:

{
  "type": "usage_checkpoint",
  "request_id": "req_20260830_example",
  "model": "deepseek-v4-pro",
  "pricing_version": "a42d372ccf0b5dd13ecf71203521f9d2",
  "effective_group": "default",
  "input_tokens_estimate": 184000,
  "output_tokens_seen_by_client": 1400,
  "stream_state": "client_open",
  "checked_at": "2026-08-30T14:11:24Z"
}

The important part is that the checkpoint belongs to the stream, not only to the gateway log. Application code can use it for three practical decisions.

First, a retry policy can distinguish "no useful output arrived" from "the user saw most of the answer but the final frame was lost." Those should not have the same retry behavior.

Second, a UI can show a cost or budget warning before a background agent keeps expanding a large answer. That warning does not need to expose every internal rate. It only needs enough context to explain that the stream has crossed a configured output or spend threshold.

Third, support can debug partial answers without asking engineering to dig through raw provider logs. A checkpoint record with model, pricing version, billing group, token estimates, stream state, and request id is usually enough to start the investigation while keeping prompt content out of the ticket.

For aiwave-sdk-python, this might fit as an optional callback rather than a response-field default:

def on_usage_checkpoint(event):
    usage_ledger.write(event)

stream = client.chat.completions.create(
    model="deepseek-v4-pro",
    messages=messages,
    stream=True,
    aiwave_usage_checkpoint_interval=2.0,
    aiwave_on_usage_checkpoint=on_usage_checkpoint,
)

The default could stay quiet for simple apps. Production agent runners, batch workers, and customer-facing long-context tools could enable it when they need a more defensible ledger.

Discussion question: should streaming usage checkpoints be an SDK-level feature, or should they remain entirely inside gateway logs? I lean toward SDK-level opt-in because retry behavior and user-visible cancellation both happen in the application.