Deadline-Aware Dispatch Hints for OpenAI-Compatible Agent SDKs (August 2026)

[chawkian]

One SDK feature I would like to see standardized is a small set of deadline-aware dispatch hints. The idea is not to make the SDK own the entire queue. The idea is to let application code tell an OpenAI-compatible gateway whether a request is interactive, deferrable, cache-sensitive, or batch-safe before the gateway spends tokens.

Most SDK calls today look immediate by default. A developer sets model, messages, max_tokens, maybe timeout, and the request goes out. That is fine for a chat turn. It is less fine for agent systems that mix user-facing steps, repository scans, background extraction, nightly evaluations, and retryable tool summaries behind the same client object.

For today's check, August 23, 2026, I pulled the live AIWave /api/pricing feed before writing. It returned success=true, pricing_version=a42d372ccf0b5dd13ecf71203521f9d2, and group_ratio values of default=3 and vip=1. Example VIP-baseline rows from that feed:

• deepseek-v4-flash: $0.319/M input, $0.957/M output, $0.0318/M cached input.
• deepseek-v4-pro: $0.957/M input, $2.871/M output, $0.0333/M cached input.
• glm-5: $0.775/M input, about $2.480/M output, $0.258065/M cached input.
• kimi-k3: $2.250/M input, $11.250/M output, $0.200/M cached input.
• qwen3.5-122b-a10b: $0.246575/M input and about $1.863/M output.

Those prices make a dispatch hint useful. A background request with a 30-minute deadline should not always compete with a user-visible request that needs a first token now. A cache-heavy repository pass may be cheaper and more reliable if it stays on a route that preserves cache affinity. A nightly evaluation job may be acceptable in a batch lane, while a support copilot response is not.

I would start with something explicit and boring:

{
  "dispatch_hints": {
    "latency_class": "interactive",
    "deadline_ms": 12000,
    "batch_allowed": false,
    "cache_affinity": "prefer",
    "approved_route_set": ["deepseek", "qwen", "glm"],
    "pricing_source": "https://aiwave.live/api/pricing",
    "pricing_version": "a42d372ccf0b5dd13ecf71203521f9d2"
  }
}

For a background job, the same shape might use latency_class="deferred", a much longer deadline, and batch_allowed=true. The important part is that the caller says what kind of work this is. The gateway can still make the final route decision because it has fresher capacity and billing context, but the SDK has preserved product intent.

I would not make these hints invisible defaults. If an SDK silently labels everything as interactive, the gateway loses the ability to protect user-facing traffic. If it silently labels everything as deferrable, user experience breaks. The hints should be visible in tests, logs, and request fixtures.

The ledger should also record the result. At minimum: latency_class, deadline_ms, batch_allowed, cache_affinity, selected model, pricing version, estimated input cost, estimated output cap, actual usage, and decision reason. That turns dispatch into something engineers can review after an incident instead of a hidden queue behavior.

There is a subtle product question here: should deadline hints be advisory or enforceable? My bias is to support both modes. In advisory mode, the gateway may override a hint during capacity pressure but must log why. In enforceable mode, missing a deadline or crossing an approved route set should return a structured error instead of quietly taking a slower or more expensive path.

Discussion question: what dispatch hints belong in the Python SDK surface, and which ones should stay gateway-only? I would include latency class, deadline, batch eligibility, cache affinity, and approved route set. I would keep provider-specific queue internals out of the SDK so applications can move between OpenAI-compatible gateways without rewriting scheduling policy.