Chinese/Unicode identifiers cause silent node deduplication — only 8 of 202 functions captured

Bug: Chinese/Unicode function names cause massive node ID collisions

Problem

The _make_id() function in extract.py (line 35) uses re.sub(r"[^a-zA-Z0-9]+", "_", combined) which strips all non-ASCII characters from node IDs. When processing files with Chinese (or other Unicode) function names, all non-ASCII characters are removed, leaving empty strings or just digits as identifiers. This causes massive silent deduplication — the graph ends up missing the vast majority of functions.

Impact

On a Lua file with Chinese function names (17,000 lines, 828KB):

Function Name	Generated ID (broken)
`道具处理类:初始化`	`""` (empty string)
`道具处理类:附灵洗练1`	`"1"`
`道具处理类:数据处理`	`""` (duplicate of above!)
`道具处理类:附灵洗练2`	`"2"`

Only 8 of 202 functions were captured in the knowledge graph.
194 functions were silently deduplicated due to ID collisions.
Tree-sitter correctly parses ALL functions — the bug is purely in graphify's ID generation layer.

Root Cause

Two locations use the same ASCII-only regex pattern:

graphify/extract.py line 35 — _make_id() uses re.sub(r"[^a-zA-Z0-9]+", "_", combined)
graphify/build.py line 38 — _normalize_id() uses the same [^a-zA-Z0-9]+ pattern

Both strip every non-ASCII character, causing Chinese, Japanese, Korean, Arabic, and other Unicode identifiers to collapse into empty strings or bare digits.

Suggested Fix

Change the regex from [^a-zA-Z0-9]+ to [^\w]+ with flags=re.UNICODE:

# extract.py _make_id():
cleaned = re.sub(r"[^\w]+", "_", combined, flags=re.UNICODE)

# build.py _normalize_id():
cleaned = re.sub(r"[^\w]+", "_", s, flags=re.UNICODE)

This preserves Unicode word characters (CJK ideographs, accented Latin, etc.) while still replacing punctuation and whitespace with underscores.

Before/After Comparison

Function Name	Before (broken)	After (fixed)
`道具处理类:初始化`	`""`	`"道具处理类_初始化"`
`道具处理类:附灵洗练1`	`"1"`	`"道具处理类_附灵洗练1"`
`道具处理类:数据处理`	`""` (duplicate!)	`"道具处理类_数据处理"`

Verification

After applying the patch locally, extraction went from 8 nodes to 204 nodes on the same file (道具处理类.lua, 828KB, 17,000 lines).

Environment

Python 3.14
Windows 11
graphify latest
Processing a Chinese Lua game server codebase

fix Graphify-Labs#811 Graphify-Labs#803 Graphify-Labs#801 Graphify-La…

Same bug persists in two more functions after this fix

The #811 fix correctly changed _make_id() and _normalize_id() to use [^\w]+ with re.UNICODE. But the same ASCII-only regex pattern still exists in two dedup normalization functions:

`dedup.py:18` — `_norm()`

def _norm(label: str) -> str:
    return re.sub(r"[^a-z0-9]+", " ", label.lower()).strip()

`build.py:184` — `_norm_label()`

def _norm_label(label: str) -> str:
    return re.sub(r"[^a-z0-9 ]", "", label.lower()).strip()

Impact

Both are called during build_merge() → build() → deduplicate_entities() and deduplicate_by_label(). On CJK labels they normalize to empty strings and are silently skipped by the if key: guard, so Chinese/Japanese/Korean nodes are never deduplicated even when they share the exact same label.

Real-world result on an 8-domain Chinese retail ERP corpus (219 nodes, --update x8):

Label	Copies	IDs
配货通知单	3	`entity_peihuo_tongzhi`, `doc_peihuo_tongzhi`, `distribution_notification`
采购合同	3	`采购合同`, `proc_contract`, `01术语参考_采购合同`
采购订单	3	...
采购入库单	3	...
		12 duplicate label groups, 28 total nodes affected

Fix

Same regex change, applied to both functions:

def _norm(label: str) -> str:
    return re.sub(r"[^\w]+", " ", label.casefold(), flags=re.UNICODE).strip()

def _norm_label(label: str) -> str:
    return re.sub(r"[^\w ]", "", label.casefold(), flags=re.UNICODE).strip()

With this, _norm("配货通知单") → "配货通知单" (preserved), the exact-match dedup pass works, and the 12 groups collapse to single nodes with merged edges.

extract/build: lap-29 cherry-pick Graphify-Labs#811 — NFKC-normalize …

fix: CJK/Unicode labels silently skipped in _norm/_norm_label dedup (…

Unify node-ID normalization into a single source of truth (#1)

Guard the LLM-facing node-ID spec against drift from the ID functions

Guard the LLM-facing node-ID spec against drift from the ID functions (…

fix(merge-graphs): normalise MultiGraph inputs to Graph before nx.com…

Unify node-ID normalization into a single source of truth (#1378)

fix Graphify-Labs#811 Graphify-Labs#803 Graphify-Labs#801 Graphify-La…

fix: CJK/Unicode labels silently skipped in _norm/_norm_label dedup (…

Unify node-ID normalization into a single source of truth (Graphify-L…

fix(csharp): emit member nodes for properties only, not fields

fix Graphify-Labs#811 Graphify-Labs#803 Graphify-Labs#801 Graphify-La…

Navigation Menu

Repository navigation

Chinese/Unicode identifiers cause silent node deduplication — only 8 of 202 functions captured #811

Bug: Chinese/Unicode function names cause massive node ID collisions

Problem

Impact

Root Cause

Suggested Fix

Before/After Comparison

Verification

Environment

Same bug persists in two more functions after this fix

`dedup.py:18` — `_norm()`

`build.py:184` — `_norm_label()`

Impact

Fix

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Chinese/Unicode identifiers cause silent node deduplication — only 8 of 202 functions captured #811

Description

Bug: Chinese/Unicode function names cause massive node ID collisions

Problem

Impact

Root Cause

Suggested Fix

Before/After Comparison

Verification

Environment

Activity

szgnewGh commented on May 13, 2026

Same bug persists in two more functions after this fix

dedup.py:18 — _norm()

build.py:184 — _norm_label()

Impact

Fix

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`dedup.py:18` — `_norm()`

`build.py:184` — `_norm_label()`