Repository navigation
atlas ingestion program: semi-automated updates, versioning, provenance, feedback — long-term design + tasks (2026-11-04 brief) #6018
Description
Program (Drew, 2026-11-04)
A long-term plan for semi-automating updates to Atlas ingestion: what gets ingested, how content is versioned, a standard provenance model, delta-only updates, retrieval statistics + result feedback, classification safety, and disk/IO + plan-spend discipline. Statement inventory (S1–S40, auditable, with per-statement dispositions): ~/workspace/scratch/atlas-ingestion-brief-2026-11-04/statement-inventory.md in the sandbox workspace; this issue is the canonical tracker — children are linked in a comment below and closed into this issue as they complete.
Where Atlas stands (2026-11-04, verified)
- Shard restoration complete: 31 TursoDB shards; internal = 28 shards / 107,001 chunks; secret = exactly 1 shard / 638 chunks (secret material only in the 2 secret DBs). Production monolith stays libSQL for now; TursoDB for all NEW databases. Cold tiers → rustfs (2026-11-04 storage ruling). Current spend anchor: Turso scaler $24.92/mo yearly — the plan is the ceiling (see also #5468, mirror storage pressure).
- Search: fan-out over shards #5755; query front-end #5773; full parameter surface with per-run vector logging #5774; Jaccard rescore + ingest dedup ≥0.9 recorded edge #5733; staging tier with merge-on-close promotion #5669.
- Provenance registry already designed: #3125 (chunk_occurrences, content-addressed extracts,
repo@commit:pathpinning). This program extends atlas: provenance registry (chunk occurrences), content-addressed extracts, multi-DB vector resolution #3125 (source-system marks, version lineage) rather than replacing it. - Scripts migration into code + fleet binary: #5670. Untracked Atlas material enumeration: #5981 — ingestion source enumeration overlaps it.
Topic map (children filed under this tracker)
- Ingestion operations design — semi-automated updates; automatic source set (reboots/boot records, session logs, chat logs, statistics, reports); usefulness filter; disk + I/O monitoring; plan-spend ceiling.
- Classification revalidation (task) — verify nothing sensitive sits at the wrong tier.
- Agent notes placement design — zettel vs repo vs publishing directly into the ID graph (ties to #5545, #5686).
- GitHub corpus extraction (task) — ALL issue/PR comments from ALL our repos; external repos we interact with or rely on (external half gated, see 5).
- External-content licensing gate (research) — may we copy external GitHub comments/discussions into Atlas at all? In the #5977/#5986 family. Must be answered before external-repo ingestion is built; design may proceed.
- First-pass corpus (task) — all code repos, all findable docs, dezren39/cairn.
- Content versioning + retention design — new versions, rename lineage, newest-resolution ("take the max"), cold rollout, deletion triggers (date / superseding increments / never accessed).
- Retrieval statistics + feedback + query trace design — top-K shard survival, top-1/2/3/5 after merge+rescore, thumbs up/down minimum, optional comments/essays, result↔query trace (per-stage scores, shard, exact embed state, metadata).
- Provenance standard — extensions filed as a comment on #3125: one standard for when/where; marks of the form
repo@hash:pathAND source-system marks (e.g.opencode-db @ amd @ <date>); segment search to associate unlineaged files; unassociated/duplicate as a logged state. - Existing-embed provenance audit (task) — scan all current embeds for correct path/provenance details.
- Delta-only ingestion + offset embeds design — the 21 GB rule (20.5 GB already present → only new files + edits land; no mass retagging); span-anchored embeds (content at file offsets 17–241) with search-time joins that never double merged lines.
- Expansive v2–v4 design — the larger design that pushes depth/scope beyond this brief wherever research leads.
Horizons
- v1 — first-pass corpus (own repos incl. GitHub comments, docs, cairn) + delta ingestion + provenance marking standard + embed audit + classification revalidation run.
- v2 — file versioning/retention live (cold rollout + deletion policy) + retrieval statistics (shard survival, top-1/2/3/5).
- v3 — feedback loop live (thumbs/comments/traces) feeding retention + ranking evolution; classification revalidation recurring/automated; offset-embed joins in production search.
- v4 — expansive: feedback-trained ranking, provenance-complete resolution, cross-source lineage graph, ID-graph-native corpus objects, classification-aware retrieval.
Gates
- External-repo comment ingestion is build-gated on topic 5's answer (Drew's sequencing: licensing figured out before the external build, after the design is somewhat done). Our own repos are not gated.
- Nothing in this program raises spend past the current plan without Drew's explicit call (S3).
Metadata
Metadata
Assignees
Labels
Infrastructure, deployment, and operationsTracking issue: closed when its children close, never triaged by coverageQueue task is parked (waiting on CI, time, or an event)Deliberately paused with a recorded resume conditionQueue packet kind: verify, reproduce, and route this itemQueue worker lifecycle state
Activity
developing-today commentedon Nov 5, 2026
CODE-1684
dezren39 commentedon Nov 5, 2026
Children filed (2026-11-04)
Designs
Tasks
Gate
Covered by existing issues (comments posted, no duplicates filed)
Statement inventory with per-statement dispositions:
~/workspace/scratch/atlas-ingestion-brief-2026-11-04/statement-inventory.md— 40 statements, zero unmapped.dezren39 commentedon Nov 5, 2026
Wave-2 update: research fold-in complete — 28 new children; child list extended
The prior-art research lane (7 clusters, 46 proposed follow-ups) has been folded into this program. Dispositions: 28 newly filed (below), 16 covered by existing program issues (#6019–#6029 family — incl. join semantics → #6028, lineage schema + newest-resolution → #6026, trace schema / survival report / feedback capture / normalization → #6027, scanner execution → #6021, connector inventory + I/O politeness → #6019, manifest-first delta → #6028, agent-notes placement → #6022, first-pass plan → #6025), 2 covered by pre-existing issues (provenance record → #3125, ingest-time dedup edges → #5733 — both with substantive research comments). Six program issues also received findings comments that change their designs (licensing gate answered-provisional on #6024; CDC verdict on #6028; git-records-no-renames on #6026; log-at-query-time on #6027; three-layer scan on #6021; cost model on #6019). Full map:
~/workspace/scratch/atlas-ingestion-brief-2026-11-04/wave2-map.md.New children by cluster
Delta / offset embeds (parent #6028): #6035 · #6036 · #6037 · #6038 · #6039 · #6040
Versioning / retention (parent #6026): #6041 · #6042 · #6043
Provenance: #6044 (parent: this tracker; home #3125) · #6045 (parent #6019)
Stats / feedback (parent #6027): #6046 · #6047
Classification (parent #6021): #6048 · #6049 · #6050 · #6051
Licensing gate (parent #6024): #6052 · #6053 · #6054 · #6055 · #6056
Ingestion ops / cost (parent #6019): #6057 · #6058 · #6059 · #6060 · #6061
ID graph (parent #6022): #6062
— Wave-2 research fold-in, 2026-11-04 (prior-art doc:
~/workspace/scratch/atlas-ingestion-brief-2026-11-04/prior-art-research.md)dezren39 commentedon Nov 5, 2026
🤖 Agent note · handed off
48 remaining items
dezren39 commentedon Nov 5, 2026
🤖 Agent note · handed off
dezren39 commentedon Nov 5, 2026
🤖 Agent note · handed off
dezren39 commentedon Nov 5, 2026
🤖 Agent note · handed off
dezren39 commentedon Nov 6, 2026
🤖 Agent note · handed off
dezren39 commentedon Nov 6, 2026
🤖 Agent note · handed off
dezren39 commentedon Nov 6, 2026
🤖 Agent note · handed off
dezren39 commentedon Nov 6, 2026
Agent note — task 10384 handed off (lane code-n) — AT CAP
opencodeperfleet queue lane— handing off to the laptop opencode claimant; this lane does not work the task directly.held,user/owned,do-not-merge,do-not-auto-edit).