watch._rebuild_code produces a ~4× bloated graph vs graphify . --update — AST-scans Markdown docs, then merges preserved semantic nodes on top #1915

Summary

graphify.watch._rebuild_code (invoked by the git hooks that graphify hook install generates for post-commit and post-checkout) produces a structurally different graph than the public graphify . --update CLI — bloated by ~4× on a mid-size corpus. This means every commit silently leaves an inconsistent, bloated graphify-out/ in the working tree.

Repro (this repo)

• Corpus: 369 files, 391k words (188 code + 181 docs + some SVGs). Curated semantic layer already cached in graphify-out/cache/ (243 files).
• graphify . --update --no-viz → 1907 nodes / 3939 edges / 33 hyperedges. ✅ Matches the committed graph shape.
• Post-commit hook rebuild (_rebuild_code via graphify hook install) → 8359 nodes / 9561 edges / 33 hyperedges. ❌ 4.4× the size. Also exceeds the 5000-node HTML viz limit, so graph.html gets deleted on every commit.

Root cause (as I read it)

In graphify/watch.py around lines 822–829:

detected = detect(watch_path, follow_symlinks=follow_symlinks)
code_files = [Path(f) for f in detected['files']['code']]

# Include document files that have AST extractors (e.g. .md, .mdx, .qmd)
for doc_file in detected['files'].get('document', []):
    p = Path(doc_file)
    if _get_extractor(p) is not None:
        code_files.append(p)

Every .md file with a tree-sitter grammar gets AST-extracted into the "code" side. Then the preservation logic at lines 467–485 keeps the existing semantic doc nodes on top, so the final graph contains both: the semantic concept nodes we curated AND thousands of tree-sitter Markdown paragraph/header nodes. That's the ~4× bloat.

By contrast, graphify . --update (via graphify/extract.py's incremental path) reads the semantic cache correctly, doesn't re-tree-sitter documents when they already have cached semantic extraction, and produces the shape the graph was originally built as.

Expected

Either:

1. _rebuild_code should honor the semantic cache the same way graphify . --update does — if a document has cached semantic nodes, don't AST-tree-sitter it as well.
2. Or: graphify hook install should generate hooks that call graphify . --update --no-viz instead of the internal _rebuild_code, so authors get the same graph shape from hooks as they do from manual rebuilds.

Impact

• Any repo using graphify hook install gets bloated graphs on every commit, then either commits the bloat or has to git checkout -- graphify-out/ after every commit.
• On repos where the initial graph was <5000 nodes but the bloated rebuild exceeds it, graph.html gets deleted every commit — surprising and disruptive.
• The "freshness contract" (post-commit rebuild trails the committed graph by one commit) doesn't actually hold in shape, only in file existence.

Workaround (what I ended up shipping)

Replaced the graphify-installed hook scripts with a small wrapper that calls the public CLI:

# .husky/post-commit (37 lines, was ~200)
# ...standard guards for rebase/merge/SKIP_HOOK/graph-only-commits...
nohup graphify . --update --no-viz >> "$LOG" 2>&1 &

Committed as: https://github.com/carlos-reyes-le-paliscot/esg-bowler-chart/pull/165. Also included a caveat in our CLAUDE.md explaining why we don't use graphify hook install — happy to remove that footnote once this is resolved upstream.

Environment

• graphifyy 0.9.15 (uv tool install)
• Python 3.11 (uv-managed)
• macOS 14
• Repo: ~370 files, ~390k words, mix of code + Markdown docs + committed semantic cache

Thanks for the tool — otherwise excellent, this is the only papercut we've hit.

🤖 Reported via Claude Code