graphify update/extract --incremental loses unrelated nodes on merge — no CLI flag to disable fuzzy dedup #2881

graphify update/extract --incremental loses unrelated nodes on merge — no way to disable fuzzy dedup from the CLI

Version: graphifyy 0.8.28

Summary: Running graphify update . (or graphify extract --incremental) on a repo with an existing multi-thousand-node graph reproducibly loses 5-10% of nodes that belong to files not touched by the incremental diff. Confirmed 3 times across 3 separate sessions on the same repo, same failure pattern each time.

Reproduction pattern:

1. Repo has an existing graphify-out/graph.json with ~2950 nodes.
2. A small, correctly-detected incremental diff (9-39 real files) is extracted.
3. build_merge() merges the new chunks into the existing graph.
4. The merged result has fewer total nodes than the existing graph (e.g. 2925, 2831, 2742 across three separate runs) even though the diff only adds/updates nodes — it never should shrink outside of prune_sources.
5. Diffing node IDs shows the missing nodes belong to files that were not part of the incremental diff at all (e.g. AdminPanel.tsx, LoginModal.tsx, unrelated Python builtins like date/int).

Root cause (traced in graphify/dedup.py, pass 2 — MinHash/LSH + Jaro-Winkler fuzzy matching): the fuzzy dedup pass in deduplicate_entities() runs over the combined node set (existing graph + new chunk) on every incremental merge. When merging a small diff into a large existing graph, cross-file fuzzy matches between semantically-unrelated nodes (generic type names, short identifiers, common labels) get merged together via _MERGE_THRESHOLD = 92.0 Jaro-Winkler similarity + community boost, collapsing legitimate pre-existing distinct nodes into one.

What I found investigating a workaround: build_merge() accepts dedup: bool = True and there's an unused safety net for exactly this case — build.py around the safety-check block:

# Safety check: refuse to shrink the graph silently (#479)
# Skip when dedup or prune_sources is active — shrinkage is intentional there.
if graph_path.exists() and not dedup and not prune_sources:
    ...
    raise ValueError(f"graphify: build_merge would shrink graph from {existing_n} → {new_n} nodes. ...")

This shrink guard only activates when dedup=False — i.e. the one mode that would actually protect against this bug is the one where the guard is disabled by the not dedup condition. But dedup=True is hardcoded in __main__.py's update/extract --incremental path (no --no-dedup CLI flag exists) with no way to opt out from the CLI.

Request: expose a CLI flag (--no-dedup, or scope fuzzy dedup to only run over the new/changed chunk's nodes rather than the full combined set) so incremental updates on large graphs can skip the fuzzy pass and get the shrink-guard for free. Alternatively, scoping Pass 2 candidates to only nodes touched by the current diff (plus their direct neighbors) would avoid the false-positive merges without disabling dedup entirely.

Workaround in the meantime: none via CLI — full re-extraction (graphify extract . without --incremental) avoids the merge path entirely but loses incremental speed, or a custom script calling graphify.build.build_merge(chunks, dedup=False) directly gets the shrink-guard but skips dedup for the new chunk too.