Repository navigation
atlas staging tier: versioned schema, registry, aggregator with merge-on-close promotion into three-level core #5669
Description
Implementation of §4 of the Scaler-era data architecture revision (design doc PR #5667, doc/2026-11-04T21-27-34Z_design_atlas_scaler_revision/).
Scope:
atlas_stagev1 schema DDL (items,chunks_raw,notes,kv,_schema_version) + one provisioning tool that creates a staging DB per work unit (queue task / session / epic / notes batch) and applies the schema.- Registry (internal core tier): unit id → staging DB, owner, sensitivity level, state (open/closing/merged), schema version. (Design §6 proposes the registry be born as a TursoDB
--tursodbpilot DB once the account toggle is enabled — sequencing note, not a blocker for the libSQL version.) - Aggregator (
fleet serverole, laptop): on unit close, embed new chunks with the resident Snowflake model, promote survivors into core in bounded batches (≤500 rows/txn), write the merge manifest (unit id, rows in/promoted, content hash), bump the core snapshot id per batch (consumed by the §3 result cache), branch the staging DB as archive, deregister. Partial periodic promotion for long-lived units (epics). - Core stores per design: exactly three physical sensitivity levels (public / internal-default / secret — separate DBs + credentials); within a level, scan stores per corpus + one content store per level; reclassification = verified mechanical row move (counts + hash) with a manifest log entry.
- Session-close extraction hook as the canonical staging producer (sessions write staging only; core never sees session-rate writes).
Acceptance: an end-to-end demo unit (one session) flows staging → close → promotion → searchable in core with correct ceiling behavior per level; a reclassification move round-trips counts/hashes; promotion never holds a core writer > seconds (batch bound enforced); core DDL only via maintenance path. PBT/MBT per the testing doctrine (schema-version invariants, manifest hash oracle).
Design: #5667 §4 (+ §5 sizing, §6 pilot). Parent context: hatch-personal#21.
Acceptance criteria
-
atlas_stagev1 schema DDL exists (items,chunks_raw,notes,kv,_schema_version) and one provisioning tool creates a per-unit staging DB and applies it. - Internal registry records unit id -> staging DB, owner, sensitivity level, state (open/closing/merged), and schema version, and is the lookup the aggregator uses.
- Aggregator promotion on unit close embeds new chunks, promotes survivors into core in bounded batches (<=500 rows/txn), writes a merge manifest (unit id, rows in/promoted, content hash), and bumps the core snapshot id per batch.
- Long-lived units (epics) get partial periodic promotion; closed units are branched as archive and deregistered.
- Behaviour matches design PR docs: Scaler-era Atlas/Turso data architecture revision #5667 section 4; deviations are recorded on this issue.
Activity
dezren39 commentedon Nov 4, 2026
Intake review
Intake by fleet monitor (repo-fleet-monitor), 2026-11-04 run.
dezren39 commentedon Nov 4, 2026
Drew approvals recorded (2026-11-04 ~17:10 CST):
db create --tursodb, so no dashboard toggle was needed. Two empty Turso-type pilot DBs now exist in groupgreat-star-lord, labeled as pilots, no data migrated:pilot-tursodb-registry(staging registry pilot, per §4/§6 of the revision doc, PR docs: Scaler-era Atlas/Turso data architecture revision #5667)pilot-tursodb-embedcache(shared embedding-cache pilot — see also the confirmation on atlas-search service: query-embedding + snapshot result caching, --no-cache/--wait/--expand/--raw flags #5668)Engine note for this issue's staging design: production atlas is confirmed libSQL (sqld) by three independent signals (
turso db showType=SQLite + libsql:// URL + sqlite_version() 3.47.0, vs Type=Turso + turso:// + 3.50.4 on the pilots). TursoDB exists in this org only for DBs created with--tursodb.dezren39 commentedon Nov 5, 2026
Core-tier shard restore — executed 2026-11-04 (per Drew's stop-and-redirect ruling, 18:24 CST)
The in-flight index rewrite was stopped (precondition verified here: laptop
atlas-index-rewrite.serviceinactive; hosted shadow frozen at 28,760 across two reads ~5 min apart) and the corpus was restored into fresh shards. Productionatlaswas only ever read — untouched.Engine + creation route (Drew ruling 18:28 CST: all new DBs are TursoDB)
CreateDatabaseV3Body.use_tursodb) and verified empirically:POST https://api.turso.tech/v3/organizations/{org-UUID}/databaseswith{"name": ..., "group_id": ..., "use_tursodb": true}. The path org must be the org UUID — the slug returns "organization not found".atlas-tursodb-probe, created via the API route):turso db showType=Turso,sqlite_version()= 3.50.4 (libSQL = 3.47.0),F32_BLOB(768)accepted,vector_distance_cosscan top-10 identical to production on the same 200 real chunks.vec F32_BLOB(768), fan-out + merge. Nolibsql_vector_idx(TursoDB rejects the syntax; scan-over-shards is the design regardless). TursoDB's metric set for future corpora: cosine, dot product, L2, Jaccard — this corpus is cosine by design (source index metadata).chunks_fts. Keyword search stays on the libSQL monolith until the consumer work decides the FTS answer (see atlas core consumer wiring: fleet atlas-search fan-out over the TursoDB shard core (tier ceilings, merge, --expand hydration) #5755).Shard map (tier → topic → shards; sessions atomic; ≤4,800 chunks/shard per §5.2)
Source: hosted
atlas, 107,639 chunks = 107,639 vectors (all 3,072 B; 0 missing/orphan). Sensitivity → tier: secret-bearing → secret; personal + local-only → internal (default); public = none in this corpus.Plus per-level content stores (§4.2):
atlas-core-internal-content(sessions 18,235; messages 70,987) andatlas-core-secret-content(sessions 90; messages 564). (The content message load needed one idempotent resume: the first pass dropped at 84% on a transport disconnect —RemoteDisconnectedescaping the HTTP client's retry filter — and a targeted second pass inserted only the missing ids to the exact counts above. Scan-shard loads were unaffected.) Every scan shard carries chunks + vec with DDL identical to source, plus anatlas_manifestself-description. Secret chunks exist only in the two secret DBs.Verification (all independent of the loader's own counts)
Corrections to the restore packet's premises (verified, second-sourced)
Leftovers
atlas-tursodb-probe(200 test chunks from the gate) kept as a test artifact — cleanup is Drew's call, along with the nine spike DBs (untouched).developing-today commentedon Nov 5, 2026
🤖 Agent note · handed off
dezren39 commentedon Nov 5, 2026
🤖 Agent note · picked up
dezren39 commentedon Nov 5, 2026
Queue triage — task 10103
Verdict: #5669 remains open and is not already implemented. GitHub REST confirms issue #5669 is open; the all-state PR search found no PR for it. Design PR #5667 merged at 7ddc3c0. I fast-forwarded the primary clone (already current) and checked main at b830347.
Current code evidence: main has pkgs/atlas-search and pkgs/fleet, but no atlas_stage schema, staging registry, merge manifest, or promotion worker implementation. The fleet README describes fleet serve search as the remaining serve-role stub; fleet's current role registry has queue-broker, secrets-broker, and search, but no aggregate role.
Scoped implementation path and order:
Ordering dependency: the schema/provisioner slice can start immediately. Before the aggregator slice, choose whether it is a new fleet serve aggregate role or is hosted under the existing search serve role; the current role registry has no aggregate entry. The required sequencing is now recorded here. This is an enhancement, so reproduction is not applicable; no repository code changed in this triage.
🤖 Agent note · triaged