Repository navigation
Add entity canonicalization/merge step to prevent duplicate nodes across files #207
Description
Problem
Graph construction currently treats extracted nodes as unique by technical ID, which is often file/chunk-
local.
As a result, the same real-world concept/entity can appear as multiple graph nodes, causing fragmentation and
noisy analytics.
Symptoms
- Duplicate semantic nodes with different IDs.
- Alias/casing/punctuation variants not merged.
- Related neighborhoods split across disconnected components.
- Distorted centrality, clustering, and query results.
Expected behavior
Graph should support a canonical entity layer so semantically identical nodes are merged into one canonical
node while preserving provenance.
Proposed solution
Introduce an Entity Resolution phase after extraction and before clustering/export:
- Normalize candidate node labels (casefold, unicode normalization, punctuation/whitespace cleanup).
- Resolve aliases via configurable dictionary/rules.
- Optionally compute similarity-based candidates with conservative thresholds.
- Build canonical IDs and merge equivalent nodes.
- Rewire edges/hyperedges to canonical nodes.
- Preserve provenance:
- original node IDs
- original labels/aliases
- source files/locations
- Recompute graph metrics (communities, centrality, report outputs) on canonical graph.
Suggested implementation details
- New module:
entity_resolution(or similar). - New config section for alias rules and thresholds.
- Optional exported artifact:
entity_map.jsonmapping original IDs -> canonical IDs. - Include merge statistics in run report (nodes merged, aliases applied, unresolved candidates)
Activity
yotamleo commentedon Aug 23, 2026
Adding a concrete data point for the cross-file half of this, from a 14,505-note Obsidian vault extracted with 0.9.48.
Because node identity is minted per source file, the same real-world entity ends up as several nodes whenever the surface form varies at all across files. Counts from one run:
@handlevs barehandle— i.e. the only difference is a leading@.@prefix (e.g.@cyrilxbt/cyrilXBT/@cyrilXBT).[[wikilink]]resolution becomes ambiguous, so the human-facing links break too.The existing fuzzy-dedup pass does not catch any of these, because it operates within a single file's extraction result — every one of these duplicates is a cross-file collision by construction.
Two observations that might make an Entity Resolution phase cheaper to scope than it looks:
@would have collapsed all 39 twin pairs and all 12 groups on its own. A conservative deterministic normalization step (no fuzzy matching, no embeddings) already captures the bulk of the damage — the expensive similarity machinery could stay opt-in for the long tail.## Connectionsedge list, zero prose), while the hand-curated note carried all the real content. "Prefer the node whose source has non-generated body content" picked correctly in 12/12 groups.Related: #296 covers the same mechanism from the node-bloat angle.