Repository navigation
atlas: provenance registry (chunk occurrences), content-addressed extracts, multi-DB vector resolution #3125
Description
Direction (Drew, 2026-10-31)
Atlas moves from a single canonical DB + one Turso publish to provenance-first, scalable infrastructure. Design doc: hatch-personal workspace/turso-atlas/docs/atlas-provenance-and-scale-design-2026-10-31.md.
Rules:
- Keep ALL data — nothing stripped from atlas.db or publishes to save space; only byte-identical dupes are cleaned, backups kept.
- Backups AND plain extracts for every corpus.
- Every chunk records EVERY place it was found:
chunk_occurrences(chunk → array of source locators with spans, observed_at). Today dedup silently drops occurrences (~127K at first ingest); 6,344 canonical sessions own zero chunks for exactly this reason and their occurrences are unrecorded. - Every import is interned content-addressed (laptop extracts CAS / R2, registry maps CA hash → normal paths) or pinned as
repo@commit:path. Registry also holds line numbers / dates / spans for extracts. - Retrieval: vector/FTS search across multiple Turso shard DBs (collection × month) returns IDs; a catalog/registry resolves an ID to a file, a chunk, or a start–end section — including a member of a tar in R2 (sidecar index, range fetch) or a local file.
Verified state feeding this (2026-10-31): canonical 18,325 sessions / 122,124 messages / 107,639 chunks; local-only = 6,941 chunks of personal-client TEXT held back from Turso by policy (not missing embeddings); Turso atlas 1.0 GB / 100,698 chunks; laptop ~/atlas 12 GB contains NO consequential byte-identical duplicates (hash-verified) — it is distinct versions/working sets, so no cleanup windfall exists.
Phases
- Add
chunk_occurrences+cas_registryto canonical atlas.db; ingest writes occurrences from now on (no re-embed, no republish). - Laptop extracts dir + CAS; atlas-backup covers it; backfill extracts for sources still on disk.
- Occurrence backfill from CAS extracts; document the local-only publish predicate in the publish script.
- Catalog DB + shard layout at next publish growth (or full message-archive ingest); resolution as MCP tools (search → ids → resolve → fetch span).
Acceptance criteria
- A
chunk_occurrencesprovenance registry exists: every chunk records every source locator it was found at (with spans and observed_at), including occurrences previously dropped by dedup. - Backups AND plain extracts exist for every corpus; imports are interned content-addressed (CAS hash → path registry) or pinned as
repo@commit:path, with line numbers / dates / spans recorded for extracts. - Retrieval works across multiple Turso shard DBs (collection × month): vector/FTS search returns IDs and the catalog/registry resolves an ID to a file, chunk, or start–end section (including a tar member in R2 via sidecar index + range fetch, or a local file).
- No data is stripped to save space; only byte-identical duplicates are cleaned, with backups kept.
Activity
dezren39 commentedon Oct 31, 2026
Update — privacy model changed (Drew, 2026-10-31)
The local-only publish gate is lifted: everything in canonical publishes to Turso, including the six personal clients (6,941 chunks). Privacy moves to query-time classification filters: scheme-1.0 labels (already complete for all 18,325 sessions in
classification-full-v1.json: personal 16,009 / local-only 2,226 / secret-bearing 90) ship inside the publish with a denormalizedsensitivitycolumn on chunks + sessions for pre-filtering, and Turso access stays restricted to trusted workers (per-DB tokens).secret-bearingis deny-by-default for any caller not explicitly granted that tier.Publish v2 build is in flight on the laptop: all 18,325 sessions / 122,124 messages / 107,639 chunks + labels + FTS + bge-small vectors for the previously held-back 6,941 chunks. It stops before the live Turso replacement, which waits on Drew's go with the v2 size + secret-bearing footprint reported. Design doc updated: hatch-personal
workspace/turso-atlas/docs/atlas-provenance-and-scale-design-2026-10-31.md.dezren39 commentedon Oct 31, 2026
Intake review
Fleet monitor intake pass (2026-10-31).
Related: design doc referenced in the body (hatch-personal
workspace/turso-atlas/docs/atlas-provenance-and-scale-design-2026-10-31.md). No merged PR or commit found that implements this — verified live, not already solved.Notes: This is a Drew-directed design/epic, not a single fix: the five rules in the body are the spec. It arrived with no labels and no acceptance criteria; both are now set (criteria derived from the body's own rules).
Risks: Scope is large (registry schema, CAS interning, multi-DB retrieval) — a single queue task should not try to land it whole. Backfill of ~127K dropped occurrences touches the canonical atlas data; rule 1 (keep all data, backups kept) must gate any migration.
Suggested approach: Split into implementation phases under this issue (registry + occurrence backfill first, then CAS interning, then multi-shard retrieval), each routed as its own task; keep this issue as the tracking epic (consider
meta/epic).dezren39 commentedon Oct 31, 2026
atlas-search CLI shipped (hatch-personal PR https://github.com/dezren39/hatch-personal/pull/9, merged 2026-10-31): hybrid FTS+vector search + get-chunk over the live v2 publish, turso and local backends, sensitivity ceiling default personal. Verified on both backends with programmatic asserts (gold-chunk paraphrase rank 1, filter correctness, sensitivity gate, provenance).
Findings for the v3 (fused-rerank) publish, from building against the live DB:
vector_top_kdoes not work on the current plain-BLOB vec table (no libsql vector index) — the CLI usesORDER BY vector_distance_cos(vec, vector32(?))(~4s unfiltered full scan). If v3 wants indexed search, publish the vec column with a libsql vector index (diskann) and the CLI can switch.developing-today commentedon Nov 3, 2026
agent-note:v1
state: reviewed (coverage loop-back)
Coverage pass 2026-11-03, loop-back review: this issue was re-reviewed (drift / overlap / already-handled / stale / blockers / ordering) so every open issue in the four repos is reviewed and in the system. Verdict: still live; in the system as task 7378 (triage, queued, priority 50) in repo-health-code-pilot. Labels audited and synced. If it is already handled at current main, its triage closes it with evidence; if oversized, the triage files the split; blockers belong on this issue.
Found via: coverage loop-back 2026-11-03
9 remaining items
dezren39 commentedon Nov 4, 2026
🤖 Agent note · handed off
dezren39 commentedon Nov 4, 2026
agent-note:v1
event: picked up
task: 7378 (triage; queue repo-health-code-pilot; worker repo-worker-code; lane c8b9355e-319a-4640-8096-7ad19480d151 — OpenChamber scheduled lane)
by: repo-worker-code
Fresh verification: issue #3125 is OPEN, assigned to Drew, and has no hard-stop labels. This is a multi-phase Atlas provenance/scaling epic, not a code-sized fix. Existing issue history says its requirements are sourced from the hatch-personal Atlas design; prior queue handoffs expired, so I am resuming from the issue/PR state rather than assuming work was done.
next: check current repo PRs/queue ownership and the linked Atlas project state; identify what is in flight, what is landed, and the record semantics/blockers before routing.
dezren39 commentedon Nov 4, 2026
agent-note:v1
event: done — triaged
task: 7378 (triage; queue repo-health-code-pilot; worker repo-worker-code; lane c8b9355e-319a-4640-8096-7ad19480d151 — OpenChamber scheduled lane)
by: repo-worker-code
Fresh state: #3125 remains OPEN; no matching implementation PR exists in developing-today/code. Atlas is still in
dezren39/hatch-personalper hatch-personal#2 (no migration scheduled). Current Atlas activity there is search/production work: PRs #9, #17–#19, and #22 are merged; no Atlas PR is open; issues #20 (writer stall/index) and #21 (search improvements) remain open. None implements the provenance registry.The adopted design doc gives record semantics: stable
source_roots; separate repo owner/name and commit SHA; virtualitem_path; eachchunk_occurrencesrow identifies(chunk_id, root_id, item_path, span_start)with spans/role/observed_at; duplicate hash hits add occurrences; CAS locators remain immutable; unrecoverable occurrences are recorded only when logs prove they existed.To keep implementation in the correct repo and make the oversized plan executable, created and verified these hatch-personal follow-ups:
No code was changed and no implementation was claimed. Queue task 7378 completed as
triaged; sandbox backlog/sweep bookkeeping is deferred to fleet reconciliation.dezren39 commentedon Nov 5, 2026
Extension from the 2026-11-04 Atlas ingestion brief — provenance standard, widened
Drew's 2026-11-04 brief (program tracker: #6018) makes this issue's provenance registry the foundation and adds requirements that extend its scope. Recorded here rather than in a duplicate issue, per the brief's dedup discipline:
repo@commit:path(S33): Drew's example — a chunk's origin mark may read "OpenCode DB @ AMD laptop on this day": the mark names system + machine + date when there is no repo. Concretely the locator grammar needs a source-system form (e.g.opencode-db@amd:2026-11-04+ session/message ids) next to the git form, so both are parseable by the same tooling.The existing-embed audit task (#6029) measures the current corpus against this widened standard and will size the backfill this registry needs.
dezren39 commentedon Nov 5, 2026
Wave-2 research addition: the PROV-lite field set for the provenance record
The Atlas prior-art pass (Cluster 3) grounds this registry's record design in W3C PROV — two of its relations are literally the version-lineage vocabulary:
wasRevisionOf(new file version ↔ prior version) andspecializationOf(a version ↔ the timeless logical file), alongsidewasGeneratedBy,used,wasDerivedFrom,wasAttributedTo. The recommended record is PROV-lite — roughly eight fields, standard names and directions, no full PROV-O/RDF deployment (a named dead end: practitioners converge on "learn the vocabulary, skip the ontology"):entity id · derived-from ids · generated-by run id · attributed-to agent · timestamps · source mark · content hash · span offsets.
Complementary shapes from the same cluster:
sourceCodeLocation/version/nominalTime/parentfacets) is the model for the ingest-run record — the run-level companion to this per-chunk registry.repo@commit:path;system@host@date;chat@session@timestamp; content hash riding in the mark) is filed as #6044 under the program tracker #6018 — this registry is its implementation home, and the existing-embed audit (#6029) is its consumer.Filed under: Atlas ingestion program #6018, wave-2 research fold-in (research item 15).
— Wave-2 research fold-in, 2026-11-04 (prior-art doc:
~/workspace/scratch/atlas-ingestion-brief-2026-11-04/prior-art-research.md)