#479 shrink-guard compares TOTAL node count, so an AST-layer gain can mask total collapse of the semantic layer — and its 'you deleted files' message misreads a dedup win as data loss

Summary

The #479 shrink-guard in to_json() refuses to overwrite graph.json when the
new graph has fewer total nodes. But a graph built through SKILL.md has two
layers produced by completely independent paths — AST (deterministic, free) and
semantic (LLM or Agent-tool subagents, expensive) — and a guard on the sum cannot
protect either one. A gain in the AST layer will mask a total collapse of the
semantic layer, and the write goes through reporting success.

Why this matters

The two layers do not fail together, and they cost wildly different amounts to
produce. On our most recent full re-extraction the AST layer grew from 18,079 to
19,938 nodes. Had the semantic pass returned nothing at all — every subagent
dying on the output ceiling described in #1758, say, which is a real and
observed failure mode — the merged total would still have comfortably cleared
the previous graph's node count. The guard would have written a graph.json
that had silently lost the entire layer that took millions of tokens to build,
and reported nothing wrong.

We didn't hit that, but only by luck of arithmetic: the guard fired on our run
for a reason unrelated to the risk it exists to protect against (the AST layer
also shrank, from deduplication).

Second problem: the message points at the wrong conclusion

When the guard does fire, it says:

ERROR: refused to shrink graphify-out/graph.json (existing graph has more nodes; #479).
If this shrink is intentional (you deleted files), re-run a full build with --force.

Deleted files is offered as the explanation for a shrink. Our shrink had no
deleted files. It was two opposite effects netting out, and the larger one was
a bug fix:

layer	before	after	cause
code	14,568	13,209	duplicate path-base copies collapsing — the fix
doc	3,505	2,061	composition inverted, not lost

The code layer shrank because a full re-extraction gave every node a canonical
ID, collapsing duplicates that a previous extractor had created under a second
path base (absolute-path IDs 1,486 → 92; one class appearing as 9 separate nodes
→ 1). That is precisely the outcome you want, and the guard's wording steers the
operator away from recognising their own fix.

The doc layer's shrink was likewise an improvement — its composition inverted:

`file_type`	before	after
`document`	3,162	114
`concept`	339	1,657
`rationale`	4	290

with INFERRED links 52 → 315 and doc↔code bridge links ~15 → 140. The old bulk
was per-heading fragments; the new layer is fewer, denser, better-connected
nodes. Node count went down, information went up.

Third problem (subtle): where you assert matters

Our own gate had this bug too, which is how we noticed it. We asserted a
doc-layer floor against .graphify_extract.json — the pre-build extraction —
and it passed at 2,167 against a floor of 2,103. Then build_from_json deduped
22,085 → 15,270 and the built graph's doc layer came in at 2,061, under our
own floor, silently. The extraction's node count is not the shipped node count.

Suggested fix

Assert per-layer, not on the total. Before writing, compare the previous
graph's per-file_type counts to the new one's and fail on any layer that
has an independent production path, rather than on the sum.
Measure the built graph, not the extraction. build_from_json dedups, so
any assertion has to run after the build. Building to a scratch path and
comparing before writing is cheap and makes the check honest.
Widen the message. (you deleted files) → (e.g. you deleted files, or duplicate nodes collapsed). The second cause is what a canonical-ID fix
produces, and it will become more common as ID normalisation improves.
Consider reporting composition on a refusal. Printing the per-file_type
delta alongside the refusal would let the operator see in one line whether
they are looking at data loss or a dedup win. Right now both look identical:
one number went down.

The general shape

A size guard can only ever ask did this shrink? — never did this get worse?,
which is the question that actually matters. "Fewer" and "worse" are independent
axes. For a merged artifact whose layers can move in opposite directions, the
aggregate is exactly the statistic that hides the failure.

Relation to existing issues

Prevent partial-chunk overwrite in --update: pre-build node-count assertion + build_merge() helper #479 — the issue that introduced the guard. This isn't a regression in it;
it's that a total-node-count assertion was always the wrong granularity for a
two-layer artifact.
_rebuild_code indexes markdown by AST but the /graphify skill does not — the two corpora fight over the shrink-guard #2089 — also about the shrink-guard, but a different mechanism: two
corpora (_rebuild_code's markdown-by-AST vs the skill's) fighting over the
same guard. This report is about two layers within one build, so the fixes
are complementary rather than overlapping.
Part B2 semantic-extraction subagents silently crash past the ~64k output-token turn limit on large/deep-mode chunks; SKILL.md chunking is file-count-based, not output-size-aware #1758 — the output-ceiling crash that makes semantic-layer collapse a
realistic scenario rather than a hypothetical one. That issue is the cause
this guard should be catching and currently can't.

Environment

graphifyy 0.9.27, Windows 11
Full re-extraction (not --update), default mode
Semantic extraction via SKILL.md Part B2 Agent-tool subagents, no Gemini key
Final graph 15,270 nodes / 22,527 links, written with force=True after
verifying the reduction per-layer by hand — which is the manual work this
report is asking the guard to do

Fifth data point for this, from a merged code+doc corpus, plus a workaround that holds.

Confirming the premise on another corpus. Our root graph is 6,166 nodes / 12,564 edges and the
two tiers partition cleanly by _origin: 4,106 AST (3,837 from 174 .py) and 2,060 semantic
(2,052 from 271 .md), zero overlap by extension. The semantic half is the expensive one, and a
total-count guard cannot see it disappear — exactly as this issue argues.

We lost it twice before diagnosing it. Six hooked rebuilds on 2026-08-12 landed at 2,167–2,331
nodes against the curated 6,166; a live hook install on 2026-08-26 began the same collapse on the
first commit and was killed mid-run. In our case the shape fallback in #2843 is not the mechanism
— 0 of our 2,060 unstamped nodes carry an L<line> source_location (1,894 have an empty one, the
rest a ## heading). Ours is the blunter path: _rebuild_code calls to_json(..., force=True), so
the #479 guard never gets a vote regardless of tier detection.

The workaround, in case it is useful to others landing here. GRAPHIFY_OUT is enough to make
this structurally safe without any patch, because graphify.paths resolves it at import and
watch.py builds out = watch_path / _GRAPHIFY_OUT:

keep the curated merged graph in graphify-out/, never rebuilt automatically;
run every automatic rebuild with GRAPHIFY_OUT=graphify-out-code, a code-only graph that is
disposable and regenerable at zero LLM cost.

Measured across four pinned rebuilds (incremental and full), graphify-out/graph.json stayed at
md5 e129c5e12da05e63f8e000a8aeba20a5. The expensive tier becomes unreachable by the cheap one
rather than merely protected by a guard.

One trap worth adding to this issue. A full pinned rebuild still pulled 2,563 markdown heading
nodes into the "code-only" graph (5,121 → 7,713), because _rebuild_code appends every detected
document with an AST extractor and markdown has one. An incremental rebuild hides this — only the
changed file is re-extracted — so it only appears on post-checkout. This is #2089 arriving through
a different door. Fixing it needed the per-output build config rather than .graphifyignore, since
both graphs share a scan root:

// graphify-out-code/.graphify_build.json
{"excludes": ["*.md", "*.mdx", "*.qmd"]}

_read_build_excludes re-applies it on every rebuild and the same full rebuild then lands at 5,150
nodes with zero markdown.

On the fix direction. Agreeing with this issue's framing: the useful invariant is per-tier
accounting, not a bigger guard. A to_json that refused to write when _origin: "semantic" node
count per source file dropped without that file changing would have caught our incident, #2843's,
and the one this issue describes — all three are invisible to any total.

Real-corpus confirmation of exactly the masking described here, from an upgrade rather than a
deletion.

Upgrading 0.9.5 → 0.9.56 and rebuilding the same commit of a ~4,300-node TypeScript corpus:

	before	after
total nodes	4,281	4,734
`calls` edges	1,613	1,915
hyperedges	22	4
`rationale` nodes	85	45

The AST layer genuinely improved (the newer extractor attributes calls inside arrow callbacks,
which the old one missed), so the total node count rose by 453 and the shrink guard never
fired — while 82% of the hyperedges and half the rationale nodes were discarded in the same
write. Nothing in the run output mentioned the loss.

Two details that may be useful for the fix:

It is version-bump-specific, not per-update. A second graphify update . on the same
version and same content left rationale at 45 and hyperedges at 4 — so a per-layer guard
only needs to fire when the layer shrinks, not on every incremental run.
The semantic layer is the expensive one and the only one an AST pass cannot rebuild.
Recovering those hyperedges took a full semantic re-extraction (~840k tokens through the
host agent); graphify label and graphify update both left them at 4. A per-layer
comparison — hyperedges, and node counts split by file_type — would have caught it, where
the total-count comparison structurally cannot.

@safishamsi

checked on v8 at 35adf43. a graph that grows from 4 nodes to 7 still writes away a saved rationale id, and a new stub keeps the semantic count flat.
i was thinking of a fix that refuses that missing id unless the file was deleted but a full extract still force-writes
cc: @safishamsi

Data point from 0.9.89 (v8 at e8eb76e) on the attribution half of this issue, through the #3774/#4014 path rather than to_json.

graphify extract . --backend claude-cli --force over a docs corpus (7 markdown files, 3 .py), existing graph.json at 457 nodes:

[graphify] Deduplicated 3 node(s) (3 fuzzy).
[graphify extract] error: dedup merged 10 node(s); writing would shrink the graph from 457 nodes to 373 nodes. Refusing to overwrite .../graphify-out/graph.json. Pass --allow-dedup-shrink to write the smaller graph.

Of the 84-node drop, dedup accounts for 10. The other 74 come from the forced semantic re-extraction producing a different node set, which is expected for --force. The message still names dedup as the cause.

The arming condition makes it worse in the other direction. In cli.py the refusal only arms when _dedup_collapsed > 0; a complete run otherwise writes with force=True. So if dedup had merged 0 nodes, the same 74-node semantic drop would have been written with no message at all. The guard's firing depends on an unrelated variable, which is the core point of this issue.

A check that matches the message would compare new_n + _dedup_collapsed against existing_n - _pruned_nodes: refuse as a dedup shrink only when dedup alone explains the drop, and report a non-dedup semantic drop separately (or per tier, as proposed above).

test(watch): stamp the shrink-guard fixtures as AST nodes

This looks fixed on the v8 branch by 7bfd59ce (fix(update): refuse a saved semantic id when the total grows). Could a maintainer confirm and close?

Navigation Menu

#479 shrink-guard compares TOTAL node count, so an AST-layer gain can mask total collapse of the semantic layer — and its 'you deleted files' message misreads a dedup win as data loss #2229

Summary

Why this matters

Second problem: the message points at the wrong conclusion

Third problem (subtle): where you assert matters

Suggested fix

The general shape

Relation to existing issues

Environment

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

#479 shrink-guard compares TOTAL node count, so an AST-layer gain can mask total collapse of the semantic layer — and its 'you deleted files' message misreads a dedup win as data loss #2229

Description

Summary

Why this matters

Second problem: the message points at the wrong conclusion

Third problem (subtle): where you assert matters

Suggested fix

The general shape

Relation to existing issues

Environment

Activity

mattgdrums-cloud commented on Aug 26, 2026

febryo-sc commented on Sep 10, 2026

SrijanSriv commented on Oct 5, 2026

Stashub commented on Oct 13, 2026

omsenjalia commented on Oct 18, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions