Repository navigation
Post-extraction node canonicalization to merge duplicate concepts #296
Description
Problem
When extraction runs per-file (which is the default for large vaults), agents emit file-scoped node IDs to avoid collisions (e.g., 2024-03_silvia, journal_silvia, crm_silvia). These all refer to the same concept but appear as separate nodes. On our vault, this caused 62% node bloat (1,421 raw nodes collapsed to 548 canonical nodes on one batch).
Proposed solution
A canonicalization pass that runs after extraction and merge:
- Label normalization: Lowercase, strip path prefixes, remove known suffix patterns (e.g., folder qualifiers that agents add for disambiguation).
- Canonical ID generation: Slugify the normalized label into a stable ID (alphanumeric + underscores, length-capped).
- Node merge: All nodes sharing the same canonical ID collapse into one. Metadata is merged (source files become a list,
mention_counttracks frequency). - Edge dedup: Deduplicate edges by
(source, target, relation)triplet after remapping to canonical IDs. - Validation: Filter self-loops and any edges that became dangling after the merge.
Why this matters
Without canonicalization, the graph grows linearly with file count even when the same concepts appear repeatedly. Community detection splits what should be one cluster into many fragments. God-node analysis misses the true hubs because their mentions are scattered across aliases.
On journal-heavy corpora, we see 34% canonicalization reduction per batch (the highest of any content type, because journal entries reference the same themes with slight phrasing variations).
Scope
This could be a graphify canonicalize subcommand or a --canonicalize post-processing flag. The normalization rules would need to be configurable since different vault types have different naming conventions.
Happy to send a PR.
Activity
yotamleo commentedon Aug 23, 2026
Independent real-world confirmation of this exact mechanism, on a 14,505-note Obsidian vault.
Per-file extraction minted 39
@handle-vs-bare-name twin pairs and 12 curated-vs-importer duplicate groups — e.g.@cyrilxbt,cyrilXBT, and a hand-curated@cyrilXBT.mdall coexisting as separate nodes, differing only by case and a leading@. Two of those groups are byte-identical filename collisions across two directories, which is not just node bloat: it's a live Obsidian link-resolution ambiguity, so[[name]]resolves non-deterministically for the human reader too.Scale of the output: the importer's own output directory now holds 7,802 of the vault's 14,505 total notes (~54%).
One detail that may be useful for scoping a fix here: every non-canonical copy we inspected was a pure mechanical stub — YAML frontmatter plus a
## Connectionsedge list, with zero hand-written prose. The hand-curated note was always the one carrying real content. So a merge-on-canonical-ID pass exactly as proposed in this issue would be safe to apply automatically in our case, and a "prefer the node with non-generated body content as the merge target" heuristic would pick the right survivor every time in this corpus.Measured on 0.9.48.
yotamleo commentedon Aug 23, 2026
Follow-up to my numbers above — I went looking for which part of the current pipeline the duplicates get past, and the answer is narrower and more actionable than I expected, so it seems worth writing down here.
Measured against the shipped
graph.jsonfor that vault (47,662 nodes), i.e. afterdeduplicate_entitieshas already run.The normalization half is already solved
_norm()casefolds and collapses[\W_]+, so every variant in my examples already keys to the same string:They land in the same
norm_to_nodesbucket in Pass 1. Nothing about the string handling needs to change — the group is formed correctly and then not unioned.What actually stops the merge
I classified every surviving cross-file
_norm-collision group by which Pass 1 cross-file gate rejects it (concept-only / provenance / entropy):file_type == "concept"gatesource_file)By the
file_typemix in the group:documentonlyconcept+documentdocument+rationaleconcept+rationaleconcept+document+rationaledocument+paperrationaleonlySo it is essentially one gate, in one place — the
n.get("file_type") == "concept"filter on the cross-filemergeablelist indedup.pyPass 1. Entropy and provenance are doing almost nothing here.The part I think is the real finding
The comment on that gate gives the rationale as:
conceptis the one type meant to unify across files, whiledocumentlabels are file-anchored (#1284). That reasoning is clearly right for file nodes — twoREADME.mdin different folders are genuinely two documents, and merging them would be a bug.But in this graph none of the
documentnodes are file nodes. I reconstructed each source file's own node id (_make_id(_file_stem(source_file))) and compared: of 39,627documentnodes, 0 are the file's own node. Every one is an entity the extractor pulled out of a note — a person, a project, a concept — that inheritedfile_type: "document"from the extension of the file it was found in, not from anything about the node itself.That is why note-heavy corpora look so much worse than code repos in this issue: they don't have a canonicalization problem that code repos lack, they have the same one, but almost none of their entity nodes are typed
concept, so the cross-file merge that already exists never gets to run on them.document39,627 vsconcept4,636 in this graph.Concretely, both of these are entity nodes, in different files, and the only thing keeping them apart is the type they inherited:
Why I'm not just sending a patch
The obvious diff — extend the gate to entity (non-file-anchored)
documentnodes — is small, but it changes merge semantics for every markdown corpus, anddedup.pydoesn't currently model "is this node its own file's node" at all, so it would need either an id reconstruction that couples dedup to extractor internals or a new field set upstream at extraction. Both are your call, not mine to assume, on an issue this old.If it's useful, I'm happy to implement whichever of these you'd prefer:
document/rationaleentity nodes that are provably not the file's own node, keeping the existing entropy and provenance guards exactly as they are.conceptand only the note itself isdocument— smaller conceptually, much larger blast radius.Same gate is what #207 describes from the cross-file angle and what #1651 would subsume. Happy to be told this is already understood and just unprioritized — in that case the numbers above are at least a size estimate.
Method note: measured on 0.9.48 with graphify's own
_norm/_entropy/_is_code/_file_stem, so the classification uses the same predicates the pipeline does rather than my own re-implementation.adelaidasofia commentedon Aug 30, 2026
Thank you for measuring this. That is a much sharper diagnosis than the issue I opened.
The
file_type == "concept"gate accounts for something my own numbers could not. The 62% I reported came from a journal-heavy Obsidian vault, so on your reading almost none of those entity nodes would have been typedconcept, and the cross-file merge that already exists never got to run on them. Same gate, different corpus shape.On your three options, 1 is the one I would want: widen the gate to entity nodes that are provably not the file's own node, entropy and provenance guards untouched. It keeps merge policy conservative and needs nothing retyped upstream. 2 fixes the real modelling error but changes what every existing graph means, which is a migration rather than a patch. 3 leaves the bloat in place and moves the work to the operator.
Worth keeping in whichever option lands: your "prefer the node with non-generated body content as the merge target" heuristic. Picking the survivor is the part that decides whether this can run unattended.
Happy to test a patch against a large journal-heavy vault if that is useful to whoever picks this up.
fix(dedup): say what the entity predicate proves and what it assumes (G…
yotamleo commentedon Aug 30, 2026
Opened #3216 implementing option 1.
The predicate turned out not to need any new field or extractor coupling:
_id_prefixesalready enumerates the ID a node standing forsource_filewould carry in every spelling a stored path may take, so "is this the file's own node" is reconstructible insidededup.pyas it stands. The widened gate additionally holds backnode_kind: "heading"/"page"nodes, which keeps the repeated-section behaviour verified in #3094 intact — repeated## Decisionsheadings across sibling notes stay one node per file, while an entity mentioned in those notes now merges. One limit is stated rather than hidden: that second half rests on the producer stampingnode_kind, so an unstamped sub-file node still reads as an entity — the PR documents the contract and pins it with a test instead of guessing.Scoped to the Pass 1 exact-normalization residue only. The entropy and provenance guards are byte-untouched, and Pass 2's fuzzy
_crossfile_fileanchored_blockedis unchanged, so #1284's near-identical parallel-module boilerplate still stays blocked.The survivor heuristic I mentioned earlier is deliberately not in it —
_pick_winnerdecides on ID shape anddedup.pynever sees body content, so choosing a curated note as the merge target is its own change. Here the file's own node is held out of the merge entirely rather than made to win it.Happy to adjust the predicate if you would rather it key off something else, and happy to have it tested against a large journal-heavy vault.
fix(dedup): merge cross-file entity nodes typed by their file's exten…
fix(dedup): say what the entity predicate proves and what it assumes (#…
omsenjalia commentedon Oct 18, 2026
This looks fixed on the
v8branch by commits 2a8b97af (fix(dedup): say what the entity predicate proves and what it assumes (#296)), 124ca10c (fix(dedup): merge cross-file entity nodes typed by their file's extension (#296)). Could a maintainer confirm and close?