Repository navigation
Semantic Extraction Chunking May Miss Cross-Chunk Relationships (No Global Linking Pass) #65
Description
Semantic extraction is executed in parallel chunks (typically 20–25 files per chunk, with each image isolated into its own chunk). Each sub-agent only receives FILE_LIST for its chunk, and the prompt explicitly scopes some relationships to “in this chunk” (notably semantically_similar_to). As a result, relationships that require jointly seeing artifacts that land in different chunks can be missed.
Current state (as implemented/documented):
Chunking strategy is defined in the skill orchestration docs, not in Python code.
Non-code files are split into chunks of ~20–25; images are one-per-chunk.
There is caching to skip unchanged files, and parallel dispatch for speed/cost, but no explicit “global linking” pass to recover cross-chunk semantic edges.
Many structural cross-file edges in code are covered by the local AST extractor, but doc/paper/image semantic edges can still be chunk-limited.
Impact:
Cross-chunk semantic relationships can be under-recalled, especially:
semantically_similar_to edges (prompt-scoped to the current chunk)
Concept links between docs/papers and images when they are chunk-separated
The issue grows with the number of semantic (non-code) files:
<= 25 semantic files: usually 1 chunk, minimal impact
26–50: 2 chunks, noticeable misses
100+: multiple chunks, increasing fragmentation and missed cross-chunk edges
Many images amplify fragmentation (each image becomes its own chunk)
Expected behavior:
Preserve parallelism/cost controls while improving recall of cross-chunk semantic relationships.
Proposed improvements (options):
Smarter chunking: group files by directory/module/topic to keep related artifacts together.
Add a lightweight “global linking” second pass:
After merging per-chunk outputs, generate node summaries and ask the model to propose cross-chunk edges (e.g., conceptually_related_to, semantically_similar_to) without re-reading all raw files.
Optional controlled overlap/bridging:
Include a small overlap window between adjacent chunks or repeat key “index/overview” files across chunks.
Acceptance criteria:
A documented and/or implemented mechanism that reduces missed cross-chunk semantic edges without requiring a full re-read of the entire corpus.
Tests or a reproducible example corpus demonstrating improved cross-chunk link recall.
Activity
MatoGropetrends commentedon Apr 8, 2026
#7 would likely fix a big part of the problem, but not all of it.
If #7 is implemented as described (a post-merge local embedding pass over all nodes that adds semantically_similar_to edges), it should significantly reduce the “cross-chunk similarity edges get missed” issue, because similarity linking would become global rather than limited to “within this chunk”.
However, it would not necessarily recover other cross-chunk relationships that depend on reading the original file context, such as references, cites, rationale_for, conceptually_related_to, or chunk-scoped hyperedges. Those would still require a separate global linking / second-pass semantic step if you want high recall across chunks.
safishamsi commentedon Apr 8, 2026
Fixed in v0.3.17 — all 6 skill files now instruct chunking to group files from the same directory together, so related artifacts land in the same extraction chunk. This is a zero-cost improvement with no extra LLM calls. A full global linking pass remains on the table for a future --mode deep enhancement if recall gaps persist.