Entity dedup merges nodes across different source files, silently removing whole documents from the graph #3094

Activity

1. abhay-codes07 commented on Aug 26, 2026

   

   abhay-codes07

   on Aug 26, 2026

   Contributor

   More actions

   Checked this against 0.9.50 (the report is from 0.7.15). The cross-file heading merge no longer happens: dedup._crossfile_fileanchored_blocked (#1284) refuses a label-based merge when either node is file_type: document or rationale and the two source_files differ, which is the guard this issue asks for — document nodes are exactly the heading-derived ones, and concept is deliberately left free to unify across files.

   Concrete check: three markdown files each with ## Decisions and ## Next steps, run through extract() and then deduplicate_entities() — 9 heading nodes in, 9 out, each still attributed to its own file.

   The shrink-guard point at the end is separate and, I think, still live on the extract path; #3105 covers the case where an LLM run silently comes back smaller, and #3110/#2843 the tier-provenance half of documents disappearing on update.
2. safishamsi

   added a commit that references this issue on Sep 5, 2026

   fix(dedup): merge cross-file entity nodes typed by their file's exten…

   ...

   124ca10
3. cabgab-dev commented on Sep 23, 2026

   

   cabgab-dev

   on Sep 23, 2026

   More actions

   Independent corroboration of this exact mechanism, found 2026-08-13 on a different corpus (sibling planning/spec documents with a shared template, not session notes) — same symptom, same root cause description as this issue's write-up.

   Our case: a project with many docs/*/plans/*.md and docs/*/specs/*.md files that all reuse the same section skeleton — numbered task headings (Tarea 5, Tarea 6, ...), repeated section titles (§4 Formato del mensaje, §5 Formato del mensaje) across separate, unrelated workflow specs. deduplicate_entities pass 2 only blocks a cross-file merge when the normalized label matches exactly; the moment two sibling headings differ by a number or a word, the general Jaro-Winkler pass (_MERGE_THRESHOLD = 92.0) merges them anyway, fusing content from genuinely different documents into one node — silently, no error, node count just comes out lower than expected.

   Symptom to detect it (same as this issue's "56 documents ended up with zero nodes"): after a merge/update, if the node count went down (or up by much less than expected), diff the node IDs of graph.json before/after — don't trust the "Deduplicated N node(s)" summary line alone, since a legitimate exact-duplicate collapse and a harmful cross-file fuse both roll into that same number.

   Mitigation we've been using (library-level, no code change): call graphify.build.build_merge(..., dedup=False) explicitly instead of the CLI default — confirmed this avoids the cross-file fusion, at the cost of losing automatic real-duplicate collapsing. Filed a related-but-distinct issue today (#3774) about the CLI's default/force=True behavior making even that mitigation hard to discover, since the shrink guard that would otherwise catch this class of silent loss is bypassed by default.

   Checked against a fresh graphifyy==0.9.66 install just now: _crossfile_fileanchored_blocked (dedup.py) does guard cross-file merges for file_type in {"rationale", "document"} specifically — a narrower, already-landed point-fix — but the general Jaro-Winkler pass this issue describes still runs unconditionally for other file_types (including whatever type semantic extraction assigns to plan/spec section headings in our corpus), so the proposed fix here (extend the guard to any node with a source_location set, regardless of file_type) still looks like the right generalization.
4. jvzelst-tech commented on Sep 23, 2026

   

   jvzelst-tech

   on Sep 23, 2026

   Author

   More actions

   Thanks for checking this against current — I upgraded 0.7.15 → 0.9.66 to verify, and the fuzzy-pass half is exactly as you describe. But the issue is not fully closed: there is a second cross-file merge on the exact-normalization pass, and it does not consult _crossfile_fileanchored_blocked.

   Reproduction on 0.9.66

   import json
   from graphify.build import build_merge
   
   def n(i, label, src):
       return {"id": i, "label": label, "file_type": "document", "source_file": src,
               "source_location": None, "source_url": None, "captured_at": None,
               "author": None, "contributor": None}
   
   extraction = {"nodes": [
       n("a_decisions",  "Decisions",  "Sessions/session-A.md"),
       n("b_decisions",  "Decisions",  "Sessions/session-B.md"),
       n("c_decisions",  "Decisions",  "Sessions/session-C.md"),
       n("a_open_items", "Open items", "Sessions/session-A.md"),
       n("b_open_items", "Open items", "Sessions/session-B.md"),
   ], "edges": [], "hyperedges": [], "input_tokens": 0, "output_tokens": 0}
   
   for dedup in (True, False):
       G = build_merge([json.loads(json.dumps(extraction))], graph_path=None, dedup=dedup)
       files = sorted({d.get("source_file") for _, d in G.nodes(data=True)})
       print(f"dedup={dedup!s:5} -> {G.number_of_nodes()} nodes, {len(files)} source file(s): {files}")

   Output:

   [graphify] Deduplicated 3 node(s) (3 exact).
   dedup=True  -> 2 nodes, 1 source file(s): ['Sessions/session-A.md']
   dedup=False -> 5 nodes, 3 source file(s): ['Sessions/session-A.md', 'Sessions/session-B.md', 'Sessions/session-C.md']
   

   Three files enter, one survives. Note 3 exact — this is the exact-normalization pass, not Pass 2.

   Why the guard doesn't fire

   The cross-file residue block admits document/rationale nodes when _reads_as_file_entity(n) is True, and its comment states the intent:

   A file's own node and its headings still never merge (#1284, #3094).
Scoped to this exact-normalization pass: Pass 2's fuzzy _crossfile_fileanchored_blocked is unchanged...

   The first claim holds only when headings can be distinguished from entities, and _reads_as_file_entity distinguishes them via node_kind. Its own docstring is admirably explicit that this is an assumption rather than a proof:

   ABSENCE OF THE MARKER IS NOT PROOF OF ENTITY-NESS: a producer minting sub-file nodes without stamping node_kind ... yields structural nodes that this returns True for, and two such nodes sharing a label in different files would merge. The conservative fix is on the producer side — stamp node_kind — not a guess here about what an unstamped node meant.

   That reasoning is sound. The problem is that the producer never got the other half of the fix.

   The producer is your own skill, and it does not stamp node_kind

   I checked graphify/skill.md at HEAD: the string node_kind does not appear in it. Neither the subagent extraction schema nor the extraction spec asks for it, so every node produced by the documented LLM path is unstamped, _reads_as_file_entity returns True for all of them, and the exact-pass cross-file merge is unguarded for exactly the corpora this issue was opened about.

   The guard is therefore inert in the default configuration. It protects corpora whose producer stamps node_kind — which, as far as I can tell from the repo, is none of them.

   Suggested fix

   Either half closes it, and they are complementary:

   1. Producer side (matches the docstring's own recommendation): have the skill's extraction schema require node_kind — "heading" / "page" for structural nodes, omitted or "entity" otherwise. This is the fix the guard was written to expect.
   2. Consumer side (defence in depth): have the exact-normalization cross-file residue block consult _crossfile_fileanchored_blocked too, so an unstamped document node cannot merge across files regardless of what the producer did. Given the docstring already treats unstamped nodes as unproven, treating them as blocked rather than permitted is the conservative default.

   Without (1), the exact pass merges heading siblings across files. Without (2), any third-party producer that forgets the marker silently reintroduces this issue.

   Impact seen in practice

   On a 245-document markdown corpus of session notes, headings like Decisions, Open items and Findings recur in most files. On 0.7.15 an --update with dedup=True dropped source-file coverage from 243 to 204 — 39 documents lost every node they had. That is what led to pinning dedup=False, which on 0.9.66 I still cannot remove for this reason.

   Happy to test a patch against the corpus if that is useful.

   Environment: graphifyy 0.9.66 · Python 3.13 · macOS · markdown-only corpus, no code files.
5. Mpasha17 commented on Oct 4, 2026

   

   Mpasha17

   on Oct 4, 2026

   Last edited by Mpasha17

   Contributor

   More actions

   Hey, I'd like to take this one. I reproduced it on 0.9.75 with jvzelst-tech's script (3 files in, 1 out), so it's still happening after #4014.

   I have a fix working locally, but it has one tradeoff I'd like your OK on. Right now, nothing tells a repeated heading like Decisions apart from a real entity like a person's name, because both are unstamped document nodes. So my fix makes dedup merge document/rationale nodes across files only when they're stamped node_kind: "entity". The extraction prompts (skill and llm.py) now ask for node_kind (heading/entity), so #296's cross-file merging keeps working for new extractions.

   The catch is that graphs extracted before this won't merge entities across files until they're re-extracted, and two #296 tests need the entity stamp added. The reporter's case passes, and the full suite has no new failures. This is separate from #4019, which is about merges within one file.

   Does this direction work for you? If so, I'll open the PR. @safishamsi
6. SrijanSriv commented on Oct 4, 2026

   

   SrijanSriv

   on Oct 4, 2026

   Contributor

   More actions

   @Mpasha17 oh, i was looking in to this issue separately from #4014

   the 0.9.66 script still returns 2 nodes on 0.9.75. those five nodes are handed to build_merge with no node_kind, so a prompt change does not run on that check.

   the exact pass unions them in the cross-file step and never calls _crossfile_fileanchored_blocked. the fuzzy pass already calls that function. calling it before each cross-file union keeps all 5 nodes and all 3 source files. same-file duplicates still merge. two concept nodes in different files still merge.

   a gate that merges document and rationale nodes only when node_kind is entity also keeps those 5 nodes, and it is a different rule. a document node stamped entity would merge again under that gate. the existing block refuses it, because one side is document and the source files differ. that also changes the unstamped #296 fixtures. test_dedup_merges_crossfile_document_entity_variants has 3 nodes and would stay at 3. test_dedup_merges_crossfile_rationale_entity_variants has 2 nodes and would stay at 2. both expect 1 today.

   i would leave the skill prompt and llm.py alone in this change. fuzzy thresholds and _reads_as_file_entity stay as they are. I can open a PR for the block call.
7. Mpasha17 commented on Oct 4, 2026

   

   Mpasha17

   on Oct 4, 2026

   Contributor

   More actions

   Thanks @SrijanSriv, that makes sense, and thanks for the clear explanation. Please go ahead with the block call; I'm dropping this one.
8. SrijanSriv commented on Oct 4, 2026

   

   SrijanSriv

   on Oct 4, 2026

   Contributor

   More actions

   roger, roger!
9. safishamsi

   closed this as completedin 7e7de7don Oct 4, 2026
10. lianbeast

    added a commit that references this issue on Oct 4, 2026

    fix(dedup): keep unstamped document nodes apart across files

    ...

    This issue will close once commit 7e7de7d is merged into the 'v8' branch.

    7e7de7d