Repository navigation
design(atlas): delta-only ingestion + offset embeds — land only changes; span-anchored chunks with non-doubling joins #6028
Description
Parent: #6018 (Atlas ingestion program). Statements S35–S38 of the 2026-11-04 brief.
Requirement (Drew, in substance)
Delta-only ingestion
Re-ingesting a largely unchanged corpus must land only the deltas. Drew's numbers: a 21 GB corpus of which 20.5 GB is already in Atlas — re-running ingestion over it must NOT update every file, retag everything, or edit all metadata to carry a new date. What lands is exactly:
- all the new files, and
- the edits to the old files — edits carry a new date and repeat some overlapping data, and that overlap repetition is accepted rather than engineered away.
Untouched content is not re-embedded, not re-tagged, not re-dated. (Content-addressed landing per #3125 is the natural mechanism; this design makes "we only take the deltas" a stated, testable guarantee of the ingest path, including metadata: no mass metadata edits for unchanged objects.)
Offset / overlap embeds
The chunker need not re-chunk a changed file along fresh boundary lines. The system may accept embeds with different overlaps that are not along boundary lines, recorded as spans into the current file version. Drew's example:
An embed is accepted whose content, joined to the file's most recent version, runs from index 17 to 241 — because indices 0–16 matched a prior embed. A different embed covers another end; everything after 241 matched a different embed. Only this one diff is new.
So an edit produces a small set of span-anchored embeds covering the changed region (with whatever overlap the matcher needed), anchored by offsets into the current version, each embed knowing which prior embeds its head and tail matched.
Search-time join
A search hit shows the full paragraph — and when results are joined together, lines that were merged do not double up. The join reconstructs source text exactly once: overlapped spans between sibling embeds (and between an embed and the file version it anchors to) are deduplicated at reconstruction, so a paragraph assembled from three overlapping embeds reads as the paragraph, not as three stuttering copies.
Design must answer
- Change detection per source type: git (commit diff), plain trees (hash walk), databases/session stores (watermarks + row hashes) — and the cost ceiling for detection itself (a detection pass that reads 21 GB to save embedding 20.5 GB has only won if reads are much cheaper than embeds; say so with numbers).
- Span anchoring: offsets in what unit (bytes/chars/lines), against which version id, and how anchors survive a subsequent edit (offset drift — when does an anchored embed detach, and how is detachment detected and logged?).
- Overlap acceptance rules: minimum match run to anchor on (the 0–16 head match), maximum overlap fraction before an embed is just a duplicate (interacts with the #5733 ≥0.9 dedup edge), and occurrence recording for the matched spans (an embed's matched head is an occurrence of the prior embed's content — #3125 semantics).
- Join/reconstruction: paragraph assembly from overlapping embeds + version text; the no-double-up guarantee stated as a testable invariant; behavior when the underlying version is cold or retired (sibling versioning design).
Done looks like
A design doc: delta-detection per source with cost model, the span-anchored embed format, acceptance thresholds, join semantics with the non-doubling invariant, and a worked example using Drew's 17–241 case end to end. Implementation split into follow-ons.
Activity
developing-today commentedon Nov 5, 2026
CODE-1694
dezren39 commentedon Nov 5, 2026
Research verdict: the span model is the design — CDC is the wrong engine at embed scale
The prior-art pass (Cluster 1) settles this design's central mechanism choice:
(embed_id, start, end)(restic/borg shape); the version costs only its new spans plus a manifest row; spans are reused only when byte-identical. (c) Reconstruction: results carry their own text (a hit reads correctly alone), and the join deduplicates by interval union per file version — sort spans by start, merge overlapping/adjacent intervals, emit each byte once. That is the testable form of this issue's non-doubling invariant.Wave-2 children filed under this issue
— Wave-2 research fold-in, 2026-11-04 (prior-art doc:
~/workspace/scratch/atlas-ingestion-brief-2026-11-04/prior-art-research.md)dezren39 commentedon Nov 5, 2026
Triage — task 10395 (lane code-e, 2026-11-05)
Verdict: valid design parent, already decomposed — no implementation at current main, nothing to fix directly in this issue.
Evidence checked fresh this pass:
held,user/owned,do-not-merge,do-not-auto-editabsent). Parent tracker atlas ingestion program: semi-automated updates, versioning, provenance, feedback — long-term design + tasks (2026-11-04 brief) #6018 is OPEN.main, Atlas is the chunk-based search service inpkgs/atlas-search(corpus chunks referenced bycorpus_chunk_idagainst~/atlas/v3/corpus.db). There is no span table, no delta-only ingest path, and no offset-embed join in the tree — this design is pre-implementation, as filed.(embed_id, start, end), and search-time reconstruction by interval union per file version (each byte emitted once — the testable form of the non-doubling invariant).Routing / ordering (the follow-ons already exist — do not re-file):
(file_version_id, start, end)+ text snapshots; migration from the chunk store. Foundation; goes first.What remains for this issue itself: the consolidated design doc answering its four questions (per-source delta detection + cost ceiling, span anchoring + drift/detach, overlap acceptance thresholds, join semantics + the worked 17–241 example). That doc is the synthesis of #6035–#6040, so it is correctly sequenced after those children land their designs/measurements — writing it now would freeze thresholds the research gates are meant to set. No blocker from Drew; the ordering above is the record.
Queue: task 10395 completes as
triagedwith this routing. The children carry their own queue tasks (all labelledqueue/queuedat check time).