Hardcoded max_tokens=8192 causes truncation cascade on dense docs (3× cost overhead)

TL;DR

graphify extract's Claude backend uses max_tokens=8192 in _call_claude (graphify/llm.py). On dense Markdown corpora, output JSON regularly exceeds this cap. The adaptive-retry layer correctly catches finish_reason="length" and recursively splits the chunk — but the truncated API call is fully billed (input tokens + 8192 output tokens) and produces zero usable nodes/edges. The same content then gets re-extracted from scratch in smaller chunks.

For our docs-only repo (1,094 Markdown files, mostly entity specs and PRDs averaging 2–8 KB body content), running graphify extract on the ~74 uncached files cost $4.31 when an honest cost estimate is closer to $1.50 — roughly 3× overhead from truncated-then-discarded calls.

Reproduction

Corpus profile:

• 1,094 Markdown docs, 1 PDF, 12 images
• Cache hit rate: 93% (preflight verified)
• Genuinely uncached: ~74 files
• Files are dense PRD/entity-spec content (typical 2–8 KB body each)

Command: graphify extract docs/ --backend claude --out .

Observed: every initial chunk of 14–22 files triggers finish_reason="length" at depth 0 because the JSON output for that many dense docs blows past 8192 tokens. Adaptive retry splits them; depth-1 chunks of 7–11 files frequently truncate again. By depth 2 (chunks of 3–5 files), most succeed. A handful hit depth 3 (single-file chunks) and graphify gives up with a partial result.

Sample log lines:

[graphify] LLM returned invalid JSON, skipping chunk: Expecting value: line 1 column 28711 (char 28710)
[graphify] chunk of 19 truncated at depth 0, splitting into halves of 9 and 10
[graphify] chunk of 22 truncated at depth 0, splitting into halves of 11 and 11
[graphify] chunk of 14 truncated at depth 0, splitting into halves of 7 and 7
[graphify] chunk of 9 truncated at depth 1, splitting into halves of 4 and 5
[graphify] LLM returned invalid JSON, skipping chunk: ... (depth 2)
[graphify] LLM returned invalid JSON, skipping chunk: Expecting property name enclosed in double quotes: line 886 column 37 (char 25096)
[graphify] single-file chunk docs/README.md truncated at max_completion_tokens — partial result kept

Final result for the run:

[graphify extract] tokens: 267,749 in / 234,087 out, est. cost (~claude): $4.3146

234K output tokens is suspicious — it's nearly 1:1 with input, when extraction output should typically be 30–40% the size of input for a clean run. The ratio implies the LLM repeatedly produced 8192-token responses that were thrown away.

Why it matters

We're using graphify extract as the canonical CI publisher of the corpus graph (the workflow regenerates and commits graph.json on every merge to master, per your "graph is a build artifact" guidance). In that model:

• First run: bears the full cost of populating cache. For our corpus this is once — but the truncation overhead made it 3× more expensive than necessary.
• Subsequent runs: only re-extract changed files. If a single dense entity-spec changes, the truncation cascade fires again on that file's chunk, paying overhead per merge.

Over a year of normal docs PRs (50–100 merges that touch denser specs), the cascade overhead compounds. Each merge that hits truncation pays an extra $0.30–$1.50 above the "should-cost" of extracting the changed files cleanly.

Root cause

In graphify/llm.py, _call_claude:

def _call_claude(api_key: str, model: str, user_message: str) -> dict:
    ...
    resp = client.messages.create(
        model=model,
        max_tokens=8192,   # ← hardcoded
        ...
    )

The chunk packer (_pack_chunks_by_tokens) defaults to token_budget=60_000 for input sizing. This input budget assumes a ~10:1 input/output ratio, but for dense extraction the ratio is closer to 4:1. Result: input fits the budget cleanly, but output regularly exceeds 8192.

Sonnet 4.6 supports up to 64K output tokens; Sonnet 3.7 up to 128K. The 8192 cap is far below what the model supports.

Proposed fixes (based on my limited understanding of Graphify intents)

1. Bump default max_tokens for the claude backend (lowest-effort, biggest win)

# graphify/llm.py
BACKENDS = {
    "claude": {
        ...
        "max_tokens": 16384,   # was implicit 8192
    },
    ...
}

def _call_claude(api_key: str, model: str, user_message: str, max_tokens: int = 16384) -> dict:
    ...
    resp = client.messages.create(model=model, max_tokens=max_tokens, ...)

16K is conservative (well under model limits) and would fit the vast majority of our chunks in a single call. 32K would be even better and still safe.

2. Make max_tokens configurable

Either as a CLI flag (--max-output-tokens N) or environment variable (GRAPHIFY_MAX_OUTPUT_TOKENS). Lets users tune per corpus density.

3. Proactive chunk shrinking based on observed output size

When a chunk truncates, halve the input budget for subsequent chunks in the same run instead of just splitting that one chunk. This is a heuristic — if one dense chunk truncates, others probably will too. Avoids paying the truncation cost N times.

4. Stream + parse incrementally

Stream the LLM response, parse JSON tokens as they arrive. If the stream ends mid-string, salvage everything parsed so far instead of throwing away the entire response. This is more invasive but recovers the wasted output tokens.

Workaround we're using

Pinning graphifyy>=0.7.4, accepting the one-time cost, watching the trend on subsequent merges. We've added a CI preflight gate (check_semantic_cache against expected hit rate) that catches cache-key drift early but doesn't help with truncation.

Other observations from the run

• The [graphify extract] semantic cache: X hit / Y miss log line never appeared in our run despite 93% cache hits being confirmed externally via preflight. Possibly a stdout buffering issue under GitHub Actions runners — would help debugging if it always flushed before the first chunk fires.
• pip install graphifyy does not pull in anthropic as a hard dep. The error message is clear (Run: pip install anthropic) but listing it as an optional extra (graphifyy[claude]) would be more discoverable.

Happy to provide more data or test patches.