Streaming LLM Responses: Async Patterns for Production Python Apps

[chawkian]

Streaming is the single biggest UX win when working with LLMs. Nobody wants to stare at a spinner for 8 seconds before seeing a wall of text.

Why Stream?

• Perceived latency drops ~10x. Users see the first token in under 500ms.
• Cancellation is natural. If the output goes sideways, you bail early instead of wasting compute.
• Memory-friendly. You process chunks instead of buffering a 4K-token response.

Async Generator Pattern

import asyncio
from aiwave_sdk import AIWaveClient

async def stream_chat(prompt: str):
    client = AIWaveClient()
    async for chunk in client.chat.stream(
        model="glm-5",
        messages=[{"role": "user", "content": prompt}],
    ):
        if chunk.delta:
            print(chunk.delta, end="", flush=True)
        if chunk.usage:
            print(f"\n\nTokens: {chunk.usage.total_tokens}")

asyncio.run(stream_chat("Explain RAFT consensus in 3 sentences"))

Gotchas I've Hit

1. Backpressure is real. If your consumer is a WebSocket relay, a slow client can OOM your server. Use asyncio.Queue with a maxsize.
2. Error mid-stream. You can't just retry — the client already has partial output. Best pattern: append a > [stream interrupted, retrying...] block and continue.
3. Token counting on stream. Some providers only send usage in the final chunk. If you need per-chunk accounting, count locally with tiktoken.

SSE vs WebSocket

For browser-facing apps, Server-Sent Events beats WebSocket for one-way streaming. It's simpler, auto-reconnects, and every CDN/proxy handles it. The AIWave SDK docs cover the SSE wire format in detail.

---

What streaming patterns are you using? Anyone doing chunk-level rate limiting or cost metering? Would love to hear approaches.