#3203 unverified-semantic-shrink guard has no verification path: a legitimately edited doc that yields fewer nodes fails every incremental run when the graph is net smaller (re-run reproduces it; only --allow-partial clears it) #3412

Summary

The #3203 "unverified semantic shrink" guard arms _extraction_incomplete whenever a re-dispatched semantic source yields strictly fewer nodes than before. When the whole graph is also smaller (any legitimate deletion elsewhere), the write is refused (force=False path) and there is no way to verify the shrink: re-running reproduces it deterministically, so an unattended incremental pipeline fails night after night until a human passes --allow-partial. A doc that was edited (content hash changed) is expected to change its node count, so that case should not count as "unverified".

Environment

graphifyy 0.9.56, Linux, Python 3.13, graphify extract <repo> --backend foundry --no-viz --out <dir> (incremental, nightly), C# monorepo ~11 000 files, ~106 000 nodes.

What happened (real run, today)

1. A 4-line edit landed in .agents/skills/pd-be-crw/feedback.md (a 13 KB markdown ledger). The previous extraction of that file had produced 8 semantic nodes.
2. Incremental run: the model returned 1 node for the edited file →

   [graphify extract] semantic extraction is incomplete: unverified semantic shrink detected for '.agents/skills/pd-be-crw/feedback.md' (8 -> 1 nodes). The shrink guard stays armed for this write; pass --allow-partial to overwrite a larger existing graph anyway.
   [graphify] WARNING: new graph has 106216 nodes but existing graph.json has 106223 (net -7). Refusing to overwrite. ...
   [graphify extract] error: extraction was incomplete (an AST/semantic pass failed) and the resulting graph is smaller than the existing .../graph.json. Refusing to overwrite a complete graph with a partial one. Re-run after fixing the failures, or pass --allow-partial to overwrite anyway.
   

   Exit 1, manifest left unstamped (correct so far).
3. Immediate re-run (the "Re-run after fixing the failures" advice): the semantic cache for that file was cleared, the model was asked again and again returned 1 node → identical refusal. Nothing on the user side can "fix the failure": the file simply describes fewer entities now (or the model is consistently terser about it), and a net −7 on a 106 k graph is not a partial extraction.

The same guard class also fired last night from the sibling path (1/29 dispatched file(s) produced no nodes — model omission), where a re-run did help. The shrink variant is the problematic one because it is reproducible.

Why this hurts

_handle_unverified_semantic_shrink (cli.py ~L182) offers only two outcomes: arm the guard, or --allow-partial. On an automated incremental pipeline --allow-partial is a blunt instrument (it also stamps genuinely omitted/failed files as done), so operators end up either failing nightly or always passing it — both defeat the purpose of the guard.

Suggested fix

Treat a semantic shrink as verified (do not arm the guard) when any of these hold:

1. The source's content hash changed since the prior extraction (the file was edited → a different node count is expected). Only an unchanged file that suddenly yields fewer nodes is suspicious (pure model variance / hollow response).
2. The fresh extraction is a genuine model response (not hollow, not truncated, file present in covered_files) and the shrink is bounded, e.g. the file still yields ≥1 node — hollow/omitted files are already covered by uncovered_files and _partial_source_files.
3. Optionally: a second consecutive fresh extraction agrees (two independent responses with the same lower count = verified), so "re-run" actually becomes a fix path.

A narrower alternative: only arm the guard for the shrink when the total graph shrink exceeds the sum of the flagged files' own losses (i.e. something beyond those files was lost). In the run above the total shrink (−7) equals the file's own loss (8 → 1), so nothing else was lost and the write should proceed.

Workaround we use

Our nightly wrapper retries extract once (helps the omission variant) and, if it fails again, runs it a third time with --allow-partial, relying on our own 90 % node-count floor and on the #2543/#2927 heal re-queues to recover omitted files later. We would much rather not need --allow-partial on an automated path.