Retry Spend Envelopes for OpenAI-Compatible Agent SDKs (August 2026)

[chawkian]

One SDK control I would like to see become more common is a retry spend envelope: a small budget object that travels with a request and limits how much extra money the SDK may spend after the first provider error, timeout, or fallback decision.

Most retry settings are written as reliability settings: max_retries, timeout, backoff_ms, maybe retry_on_429. That is not enough for multi-model agent traffic. A retry against a short DeepSeek Flash call and a retry against a long Kimi or GLM planning call are different financial events. If the SDK hides that difference, a reliability improvement can quietly become a billing surprise.

For today's check, August 22, 2026, I pulled the live AIWave /api/pricing feed before writing. The snapshot returned pricing_version=a42d372ccf0b5dd13ecf71203521f9d2, with group_ratio showing default=3 and vip=1. Example rows from that live feed:

• deepseek-v4-flash: $0.319/M input, $0.957/M output, $0.0318/M cached input.
• deepseek-v4-pro: $0.957/M input, $2.871/M output, $0.0333/M cached input.
• kimi-k2.7-code: $0.945/M input, about $3.000/M output, $0.150794/M cached input.
• kimi-k2.6: $0.545/M input, about $2.300/M output, about $0.1714/M cached input.
• glm-5: $0.775/M input, about $2.480/M output, $0.258065/M cached input.
• qwen3.5-122b-a10b: $0.246575/M input and about $1.862/M output.

Those numbers should not be hard-coded into retry logic forever. They should be inputs to a dated envelope. The envelope says what the SDK is allowed to spend after the first attempt, which price source was used, and whether fallback can change model families.

I would start with a request policy like this:

{
  "retry_spend_envelope": {
    "pricing_source": "https://aiwave.live/api/pricing",
    "pricing_checked_at": "2026-08-22",
    "pricing_version": "a42d372ccf0b5dd13ecf71203521f9d2",
    "billing_group": "vip",
    "max_incremental_usd": 0.015,
    "max_attempts_after_first": 1,
    "allow_family_fallback": true,
    "allowed_fallback_families": ["deepseek", "qwen", "glm"]
  }
}

The important word is incremental. A product team may approve the original request because the user clicked a button. That does not automatically approve two more long-context attempts, a Pro fallback, and a streamed response replay. The SDK should account for the planned additional exposure before it retries.

In practice, the SDK can evaluate the envelope with four values it already knows or can estimate: input tokens, cached input tokens, output cap, and selected model row. If the first attempt fails before output starts, one bounded retry may be acceptable. If partial streamed output has already reached the client, the envelope should usually fail closed unless the caller explicitly marks the operation as replay-safe. If a fallback route would cross the incremental budget, the SDK should return a structured error instead of quietly escalating model cost.

This also improves observability. Every retry event should record attempt_index, original_model, retry_model, estimated_incremental_usd, remaining_envelope_usd, pricing_version, and decision_reason. Support can then explain why an SDK retried, why it did not retry, or why it switched from one model family to another. Finance can separate normal retry spend from routing drift. Developers can review policy changes in pull requests.

There are two traps to avoid.

First, do not make the envelope a global account setting only. Account-level limits are useful, but retry behavior depends on task type. A support summary, code patch, compliance review, and batch extraction job should not inherit the same fallback budget just because they share one API key.

Second, do not estimate against a stale or invisible rate card. If the SDK cannot name the pricing source, checked date, and group behavior, it should mark the retry decision as unaudited. That does not mean every request must fetch live prices. It means the snapshot used for the decision must be explicit.

Discussion question: should retry spend envelopes be enforced inside the Python SDK, inside the gateway, or both? My bias is both. The gateway has the freshest rate card and billing group, while the SDK owns task policy, replay safety, local tests, and developer-visible failures. What fields would you require before allowing an automatic retry or fallback?