Building Resilient LLM Pipelines: Retry Strategies and Timeout Patterns

[chawkian]

When building production-grade applications on top of LLM APIs, the happy path is easy — but what happens when latency spikes, rate limits hit, or the model returns unexpected output?

The Reality of LLM API Reliability

In my experience integrating multiple model providers, three failure modes account for ~90% of production incidents:

1. Transient network errors — TCP resets, DNS hiccups, brief 503s
2. Rate limiting (429) — Especially during traffic bursts
3. Semantic errors — The API returns 200 OK but the response is malformed or empty

A Battle-Tested Retry Pattern

Here's a Python snippet using exponential backoff with jitter that I've found works well with the AIWave API:

import asyncio, random
from aiwave_sdk import AsyncAIWave

client = AsyncAIWave()

async def robust_completion(prompt, max_retries=5):
    for attempt in range(max_retries):
        try:
            resp = await client.chat.completions.create(
                model="gpt-4o",
                messages=[{"role": "user", "content": prompt}],
                timeout=30
            )
            if not resp.choices:
                raise ValueError("Empty response")
            return resp
        except Exception as e:
            if attempt == max_retries - 1:
                raise
            wait = min(2 ** attempt + random.random(), 60)
            print(f"Retry {attempt+1} after {wait:.1f}s: {e}")
            await asyncio.sleep(wait)

asyncio.run(roburst_completion("Explain backpropagation"))

Key Takeaways

• Always set explicit timeouts — default SDK timeouts can be 60s+, way too long for user-facing apps
• Add jitter to prevent thundering herd on retry storms
• Validate response shape, not just status code — semantic errors are real
• Log retry attempts — patterns of retries reveal upstream issues

Full SDK docs and more patterns at aiwave.live.

What retry strategies have worked for your team? Has anyone tried circuit breakers (like tenacity or circuitbreaker) with LLM APIs specifically? Curious about the tradeoffs between resilience and latency.