[RFC] Native graphify update for the semantic layer — a scripted conductor with a pluggable LLM hook (we measured: the math is 0.89s–52s, the agent-walked workaround is 11–14 min) #1710

Versions: measured on graphifyy 0.8.46 (our production pin is 0.8.41); design
verified against current v8 (0.9.8) — the relevant behavior is unchanged.
Env: non-interactive CI/agent pipelines over two repos: a mid-size TypeScript
repo (14,799 nodes / 25,980 edges / 711 communities) and a large C# monorepo
(112,714 nodes / 258,776 edges / ~125 MB graph.json).

Problem

graphify update today is AST-only. __main__.py (v8: the update command,
~line 3789) calls watch._rebuild_code and then prints:

Code graph updated. For doc/paper/image changes run /graphify --update in your AI assistant.

So there is no headless command that performs a full incremental update
(semantic re-extraction of changed docs + merge + re-cluster + re-label +
re-score). Every CI/agent consumer has to drive the /graphify --update skill
through an interactive assistant session that walks the steps as LLM turns.

We run graphify as a committed knowledge graph in CI at monorepo scale, and we
measured where the time goes:

More than 90% of the wall-clock is LLM-turn overhead — the agent reading the
skill, deciding each step, writing inline Python, and (worst) re-labeling
~700 communities whose membership barely changed. Only a small slice of the
run is the actual necessary LLM work: semantic re-extraction of the changed
files. On a warm cache that slice is tiny — one of our field runs re-extracted
1 file where a cold run would have extracted 207.

This overhead is package-shape-driven, not essential: every building block the
update needs already exists as a public function. The skill exists because
there is no command composing them.

Proposal: graphify update --semantic (a scripted conductor)

A native subcommand that composes existing internals, shelling out to an LLM
only for the one step that needs it:

1. Detect — detect_incremental(root, kind="semantic") (detect.py)
   already returns exactly the new/changed files since the last manifest.
2. Cache check — check_semantic_cache(files) (cache.py) partitions
   into cached / uncached. Cache entries are already content-keyed (SHA-256 of
   file bytes) and stored under graphify-out/cache/semantic/ — see
   "extraction cache is portable" below.
3. Extract only the uncached files via a pluggable LLM hook: a
   configurable command/backend invoked per chunk (the existing headless
   backends, or claude -p/any agent CLI supplied by the user). This is the
   only metered step, and its scope is the delta, not the corpus.
4. Merge — build_merge() with the changed chunks +
   prune_sources for deletions. (Merge currently severs edges from unchanged
   files into re-extracted files — that's a separate bug report,
   [Bug] build_merge severs edges from UNCHANGED files into re-extracted files (−12% doc↔code in one update cycle): endpoint nodes are replaced, the surviving edges dangle, and the dangling-edge drop silently eats them #1711; the conductor is the natural place for the fix to land.)
5. Cluster + score — cluster() / cohesion scoring, natively (this is the
   0.89 s–52 s deterministic part).
6. Label carry-forward — match old↔new communities by member-overlap
   (Jaccard) and keep existing labels; LLM-label only genuinely new/changed
   communities in one batch. This kills the ~700-community relabel that
   dominates the agent-walked cost. This is exactly Feature: carry community labels over re-clustering via member-overlap matching (invalidation exists, carry-over does not) #1653 / PR feat(cluster): carry community LLM labels over re-clustering via member overlap #1662 — the
   conductor would consume that mechanism rather than reinvent it.
7. Report + verify — before/after node/edge/cross-edge/hyperedge counts
   and a fail-loud exit when the result is degraded (zero nodes, unexpected
   byte-identity, cross-edge erosion beyond a threshold). We've been burned
   twice by silent no-op updates in CI; a native command can make "quiet
   success" and "loud failure" first-class.

Expected result from our measurements: a warm-cache update on the 14.8k-node
repo drops from ~14 min to ~1 min (extraction of a handful of files + 0.89 s
of math + one small labeling batch), and the monorepo update's $7.89 spend
concentrates on extraction instead of conductor turns.

The extraction cache is portable — lean into it

cache/semantic/ entries are keyed by content hash (load_cached,
cache.py), independent of machine, path prefix (post-#1417), and tool
version. We commit graphify-out/cache/semantic/ to git: every fresh CI
worktree warm-starts, and a full re-index of 207 files becomes a re-extraction
of just the files whose content actually changed. The conductor design should
treat the semantic cache as a first-class, shareable artifact (it already is
one — this is documentation + a --cache-from style affordance more than new
mechanism). One layout change makes this viable at monorepo scale: the flat
cache/semantic/ directory holds ~20k entries for our corpus, which is
pathological both for filesystems and for git tree churn — that's a small,
independent change proposed separately in #1709 (two-hex prefix
subdirs, git-loose-object style).

Relation to existing issues

• Feature: recursive update command for monorepo/workspace layouts #1687 (recursive update for monorepos) — composes cleanly: --recursive
  would loop the same conductor per sub-repo.
• Feature: carry community labels over re-clustering via member-overlap matching (invalidation exists, carry-over does not) #1653 / PR feat(cluster): carry community LLM labels over re-clustering via member overlap #1662 (label carry-over via member overlap) — step 6 above; the
  conductor is a consumer of that fix.
• Non-deterministic graphs from identical inputs: extraction lacks seed/prompt-override controls and extract_corpus_parallel ignores extraction-spec #1695 (deterministic extraction controls) — orthogonal but complementary:
  a scripted conductor plus seeded extraction gets reproducible updates.
• build_merge / --update drops existing graph's hyperedges from unchanged files (only nodes+edges are read from graph.json) #1574 (hyperedge carry-forward, fixed) and [Bug] build_merge severs edges from UNCHANGED files into re-extracted files (−12% doc↔code in one update cycle): endpoint nodes are replaced, the surviving edges dangle, and the dangling-edge drop silently eats them #1711 (edge
  preservation) — the merge step's correctness issues; the conductor
  inherits whatever build_merge does.
• Headless extraction excludes code from the semantic pass (per
  how-it-works.md and the 0.8.44 changelog; the cross-domain linkage gap is
  [Feature Request] Cross-domain edge generation: AST↔semantic subgraph bridging (+ alignment solution) #1254 / Semantic layer is mostly disconnected from the AST graph #198 / Proposal: AST import-graph aware chunking (uses Part A output to inform Part B packing) #627 territory) — which is why step 3 must be pluggable:
  consumers who need co-chunked doc+code extraction can supply a richer
  agent-backed extractor while the pipeline around it stays native.

Offer

We are building exactly this downstream right now (a scripted conductor
wrapping graphify's public functions, with the cache/carry-forward/verify
behaviors above), because we couldn't keep paying 14 minutes of agent time for
1 minute of work. We'd much rather it live upstream where every CI consumer
gets it. Happy to contribute the PR — including regression tests pinned to
the numbers above — if you're open to the shape. Questions we'd want your
steer on first:

1. Extend graphify update (flag-gated) vs a new subcommand?
2. What should the LLM hook contract be — reuse the existing headless backend
   interface, or a generic "command that takes chunk JSON on stdin"?
3. Any objection to committing/sharing cache/semantic/ as a supported flow?