dedup: KeyError 'source' in deduplicate_entities when merging extraction into existing graph.json #803

Summary

graphify extract crashes during deduplication when merging newly-extracted nodes/edges into an existing graph.json. Reproducible across multiple Ollama models on different hosts.

Stack trace

File "/usr/local/lib/python3.13/site-packages/graphify/dedup.py", line 236, in deduplicate_entities
    e["source"] = remap.get(e["source"], e["source"])
                            ~^^^^^^^^^^
KeyError: 'source'

Reproduction

1. Bootstrap graphify with a small markdown vault → existing graph.json has nodes + links (NetworkX-style: each link has a source key).
2. Add new markdown files with YAML frontmatter to the vault.
3. Run graphify extract /vault --backend ollama --model qwen2.5:7b --out /data.
4. Extraction completes (cache fills under graphify-out/cache/semantic/), but the dedup step crashes with the trace above.

What I verified

• All 22 cached LLM responses under cache/semantic/ contain nodes + edges + hyperedges arrays. Every edges[i] has the keys [confidence, confidence_score, relation, source, source_file, source_location, target, weight] — none missing source. Verified with:

  find … cache/semantic -name '*.json' -exec cat {} \; \
    | jq -s '[.[].edges[] | select(has("source") | not)]'
  

  → empty array.
• The pre-existing graph.json has 38 links and all of them have source too.
• Reproducible with both qwen2.5:7b and qwen3.6:35b on separate Ollama hosts (so it's not a model-output shape issue).

Hypothesis

Somewhere between combined["edges"] construction and deduplicate_entities() (around graphify/dedup.py:226–240), an edge with no source key is being included. Two candidates:

• Hyperedge → edge promotion. If hyperedges are being flattened into the edges list during merge and the flattening doesn't always emit a source key, that would match.
• NetworkX-format links → edge format mismatch. The pre-existing graph.json stores edges under links (NetworkX convention) with source/target as node-IDs. If those are being loaded back into the dedup pipeline as "edges" but with a slightly different shape (e.g. integer IDs vs string keys, or stripped during a normalization step), one of them might lose source before the remap.get line.

A simple guard at line 236 (if "source" not in e: continue or equivalent, with a debug log of the offending edge) would both fix the crash and surface which path is producing the malformed edge.

Workaround

Reverted manifest.json to a pre-extract backup so graphify reprocesses cleanly. The upstream markdown source files themselves are produced correctly; it's only the merge into the existing graph that fails.

Environment

• graphify version: graphifyy[mcp,watch,pdf,ollama]==0.7.13
• Python: 3.13 (Docker base image)
• Backend: Ollama (qwen2.5:7b on one host, qwen3.6:35b on another — both reproduce)
• Vault: ~50 markdown files with YAML frontmatter

Happy to share a redacted cache dump or a minimal reproduction if it would help.