First-token latency is a product metric, not a model benchmark

[chawkian]

First-token latency is a product metric, not a model benchmark

Many LLM dashboards report only total request latency. For an interactive developer tool, that hides the metric users feel first: the time until the first useful token. A router can improve perceived speed without changing the model, simply by streaming correctly and avoiding unnecessary buffering.

I compared two current AIWave catalog identifiers on 2026-08-03:

Model	Input / output per 1M tokens	Practical role
deepseek-v4-flash	$0.206 / $0.412	Short answers and interactive classification
qwen3-coder-480b-a35b-instruct	$0.12 / $0.36	Code explanations and generation

Rates are a dated snapshot from AIWave pricing; refresh them before making a budget decision.

Measure the stream in three phases

For every request, record:

1. Connect time — DNS, TCP, TLS, and request upload.
2. First-token time — the interval until the first non-empty delta.content.
3. Completion time — the interval until [DONE] or a terminal error.

These measurements answer different operational questions. A high connect time points to networking or a saturated proxy. A high first-token time points to queueing or model scheduling. A long completion time may simply mean the prompt requested too many output tokens.

Minimal measurement wrapper

import time
from openai import OpenAI

client = OpenAI(base_url="https://aiwave.live/v1", api_key="YOUR_API_KEY_HERE")

started = time.perf_counter()
first_token = None
chunks = []
stream = client.chat.completions.create(
    model="deepseek-v4-flash",
    messages=[{"role": "user", "content": "Summarize this pull request in five bullets."}],
    stream=True,
    max_tokens=300,
)
for chunk in stream:
    text = chunk.choices[0].delta.content or ""
    if text and first_token is None:
        first_token = time.perf_counter()
    chunks.append(text)
finished = time.perf_counter()

print({
    "first_token_ms": round((first_token - started) * 1000, 1) if first_token else None,
    "total_ms": round((finished - started) * 1000, 1),
    "characters": len("".join(chunks)),
})

Run the same fixed prompt set repeatedly. Do not compare one short prompt on one model with a long codebase prompt on another; report p50 and p95 for each workload class.

Buffering can erase the benefit

The provider may emit small SSE events, but a reverse proxy can buffer them until a larger threshold is reached. For a streaming route, disable proxy buffering, keep the response Content-Type: text/event-stream, and send periodic heartbeats if an intermediary closes idle connections. Validate the complete path—from the provider to the browser—not only a direct server-to-server request.

Retry boundaries matter

Retrying before the first event is usually safe for a read-only completion. Retrying after partial output can duplicate text and can repeat a tool call. Use a request ID and classify failures as before_output, partial_output, or complete. For tool-using agents, store an idempotency key with every side effect so a client reconnect cannot charge or mutate state twice.

Choosing a default route

Use a fast model for predictable, bounded work and reserve a stronger model for tasks where quality justifies the extra tokens. The right decision comes from your own evaluation set: first-token p95, completion p95, error rate, output quality, and USD per successful task. Model names and prices change; keep the catalog lookup beside the measurement code and review it when the pricing page changes.

The AIWave Chat Completions documentation shows the OpenAI-compatible request shape. What is your current first-token p95, and which buffering or retry behaviour has affected it most?