_rebuild_code indexes markdown by AST but the /graphify skill does not — the two corpora fight over the shrink-guard #2089

Activity

1. LoGui42 commented on Aug 12, 2026

   

   LoGui42

   on Aug 12, 2026

   More actions

   Independent reproduction on 0.9.28, different stack (Python/ML research repo, 1,145 files in the graph, 14 subjects' worth of planning docs). Confirms the mechanism and adds a few measurements that may help, including one negative result that narrows where the write happens.

   The two corpora, measured

   path	nodes
   post-commit hook (_rebuild_code, code + doc-AST)	12,719
   CLI graphify update . (code-only corpus)	11,937
   difference	782

   Proof the hook is what wrote the larger graph — not a stale artifact:

   graph.json mtime : 2026-08-12 14:10:51
   commit 41b6535   : 2026-08-12 14:10:43     <- 8s earlier
   graph.json       : "built_at_commit": "41b65356745febd590a95f0b31d745d135e9124d"
   

   So every commit installs the 12.7k-node corpus, and every subsequent graphify update . trips the shrink-guard against it. Exactly the non-converging loop described here.

   What --force actually costs, by node

   Ran the forced rebuild against a copy of the repo + graphify-out so the real graph was never at risk, then diffed old vs new keyed on (source_file, label, source_location):

   class	nodes	verdict
   markdown heading nodes from 14 docs/*.md	312	legitimate layer, silently deleted
   stale document nodes from git-ignored files	475	correct to purge
   (my test-harness artifact — excluded data/ to skip a 390MB .h5)	317	not a real loss

   Confirms this issue's point 2: nothing in the output separates the 475 that should go from the 312 that shouldn't. Largest single victim was docs/ARCHITECTURE-AND-TRAINING-MATH.md (119 heading nodes), then docs/GATE-0-VERDICT.md (29), docs/guides/training-diagnostics.md (28).

   The 475 stale ones are surfaced separately and correctly, which is a nice touch:

   [graphify watch] fail-closed: kept 481 node(s) from 319 file(s) that left the scan corpus
   but still exist on disk (ignore rules or filters changed?).
   

   Those 319 are exactly the set "in graph.json but absent from manifest.json", and all 319 are git-ignored (306 under outputs/, 10 under a nested .superpowers/sdd/.gitignore, +3). Cross-checked with git check-ignore --stdin over every source_file in the graph: 316 hits, the small delta presumably graphify's own filters.

   Negative result: the refusal does not write

   Worth recording, since I initially suspected the guard of writing a partial merge (which would explain monotonic growth). It does not. Controlled test on a replica — refuse, then re-hash:

   BEFORE  sha256 787ae25f2d8c...  nodes 12719  mtime 23:25:11.584599728
           graphify update .   -> "Refusing to overwrite"
   AFTER   sha256 787ae25f2d8c...  nodes 12719  mtime 23:25:11.584599728
   

   Byte-identical. So the growth I saw came purely from the post-commit hook re-establishing the larger corpus, not from the guard leaking writes. If anyone else reports "the graph grows even when update refuses", the hook is the thing to look at first.

   Diagnostic wording

   The guard's message attributes the shrink to a cache problem:

   Refusing to overwrite — you may be missing chunk files from a previous session.

   On this repo the cache is genuinely thin (cache/ast/ 1 entry, cache/semantic/ 164 against 913 manifest files, 771 with an empty semantic_hash), so that reading is plausible and sent me looking for cache corruption for a while. The actual cause was the corpus difference. If the guard could compare the file sets rather than only the node counts, it could say "N files present in the existing graph were not scanned this run" — which is the fact that actually localises it.

   Small positive note

   --force does snapshot first — it created graphify-out/2026-08-12/ with 4 files before overwriting, and the real repo has four such dated backups. That makes the data loss recoverable, which is worth mentioning in the guard's message as the reason --force is safe to try.

   Workaround adopted

   Not forcing. On this repo the stale 475 point at files that still exist on disk, so queries do not dangle — they occasionally surface run artifacts, which is tolerable. Will purge only as part of a deliberate full re-extraction. Option 2 from this issue (GRAPHIFY_NO_DOC_AST / --no-doc-ast honoured by both paths) would resolve it cleanly here.
2. kanc741 commented on Oct 21, 2026

   

   kanc741

   on Oct 21, 2026

   More actions

   Same root cause via the CLI, not only the hook — still present on 0.9.93.

   Tiny repro (one .py file, two .md files): graphify extract . --code-only gives 2 nodes; after graphify update . the graph has 10 nodes, 7 of them from the .md files.

   mkdir md-repro && cd md-repro
   printf 'def a():\n    return 1\n' > app.py
   printf '# Title\n## Section A\n## Section B\n' > README.md
   mkdir docs && printf '# Guide\n## Step 1\n' > docs/guide.md
   graphify extract . --code-only        # 2 nodes, 0 from .md
   printf '\ndef b():\n    return 2\n' >> app.py
   graphify update .                     # 10 nodes, 7 from .md

   graphify update --help says "re-extract code files", and update has no --code-only option, so there is no way to keep a code-only graph code-only. #1915's fix only covers docs that already have semantic nodes, so this case (no semantic layer, code-only workflow) remains. On a real ~400-code-file repo with many docs it was +33% nodes (0.9.17). A later extract --code-only keeps those nodes, which I understand is intended since 0.9.49 preserves the document layer — so the fix point is update itself.

   Suggestion: skip the markdown quick-scan in update when the existing graph has no semantic layer / was built with --code-only, or add update --code-only.