Repository navigation
Atlas step 2: core search library (atlas-core) #5352
Description
Step 2 of the Atlas binary+service epic (#5350). Design of record: doc/2026-11-04T08-07-13Z_design_atlas_binary_service/ (§4 atlas-core, §5 Gate C).
Scope
atlas-corecrate: the search semantics of the Pythonatlas_search.py, ported faithfully —Filters→SQL construction (classification filters as pre-filters on BOTH branches), sensitivity ceiling logic (SENS_ORDER = [shareable, personal, local-only, secret-bearing], default ceilingpersonal, ceiling applied before ranking), branch query builders (FTS5/BM25 + vector), RRF fusion (k=10, pool 100 defaults), the result envelope types (results carryrank_sources+ fusedscore, matching the Python's JSON shape).- A
SearchBackendtrait with two implementations: Turso (libsql HTTPS pipeline, read-only token) and Local (SQLite publish files; side stores joined by content hash, never chunk_id). - All three vector strategies behind a
VectorBackendenum —Scan(exactvector_distance_cosover the pre-filtered allowed set),IndexFixed,IndexFill(escalating depth, cap 3200) — mirroring the Python's--vector-backendmodes. - Query embedding is consumed through a trait so fixture vectors stand in during this step (the real embedder is step 1).
Dependencies: none hard (fixture vectors stand in for step 1). The default VectorBackend is decided by the golden-311 A/B running on the Python (design Gate C): start with Scan (the incumbent); if the A/B verdict lands during this step, set the default to the winner — a one-line change, recorded on this issue.
Verification gate (exit criteria)
- Core-level tests reproduce the Python's branch/fusion behavior on fixture data: identical fused rankings (same order, same scores within float formatting) for a recorded set of Python-produced branch pools, including ceiling-filtered cases where restricted chunks must be absent from pools entirely (not merely down-ranked).
- Envelope serialization matches the Python's field-for-field on golden examples.
Out of scope: the embedder (step 1), CLI argument surface (step 3), HTTP anything (step 4).
Activity
Merge pull request #5372 from developing-today/atlas-step2-core
dezren39 commentedon Nov 4, 2026
Step 2 complete —
atlas-corelanded via PR #5372, mergeda65000aeabf0f3d18fec54cb547c135f36898090(step commit51d29a98d).What was built
pkgs/atlas-search/crates/atlas-core— the search pipeline as a library, a semantics-exact port of the production Pythonatlas_search.py, on top of step 1'satlas-embed:filters(Filters + sensitivity ladder + every SQL statement builder),fusion(FTS match expr, RRF k=10),backend(theSearchBackendtrait),turso(HTTPS/v2/pipeline, read-only token flow env → Bitwarden → 20h cache → CLI mint; never writes),local(SQLite publish files, temp-view schema normalization, content-hash side stores, in-process cosine scan),embed(query-embedder boundary),search(run_search/run_get_chunk+ envelope rendering),types,error.atlas-protocrate here: the design assigns protocol types to step 4 (Atlas step 4: serve mode — job protocol, worker pool, in-memory temp store #5354); the envelope isserde_json::Valueuntil the serve protocol shapes them.Gate evidence
round6= CPythonround()parity, hrana codecs, fetch-to-fill depths.fixtures/gen_core_fixtures.py→fixtures/core-expectations.json): all 12 search envelopes byte-equal (hybrid/fts/vector, ceilings, kind/topic/client/date filters, grouped, expand+message, pool/k truncation), both get-chunk envelopes byte-equal, Turso branch SQL byte-for-byte for 4 filter shapes, and theallowed_ids/ index-fill (depths [3, 6] at pool 3) / index-fixed call sequences exact against a recording transport. Fixture canary: the secret-bearing chunk holding the query's exact vector is absent at the default ceiling, and ranks vector test #1 (d = 0.0) at the secret-bearing ceiling.crates/atlas-core/examples/live_smoke.rs, read-only against production Turso, scan backend): "how do we back up the database", "what did we decide about the Muse subscription quota pools", "nixpkgs update workflow for the fleet", "how does the atlas search ranking work", "what happened with the cutter package build failure", "explain the reflective event archive design" — Rust fused top-10 identical to the deployed Python CLI's on every one. (Full id lists in PR atlas-core: core search library (Atlas epic #5350 step 2, #5352) #5372.)cargo test --workspacegreen,cargo clippy --workspace --all-targets0 warnings,cargo fmt --checkclean.API notes for steps 3–4
&mut selfanswer):atlas_core::AtlasEmbedderwraps the step-1Embedderin aMutex—Session::runis&mut self, so queries serialize on it. Serve mode (Atlas step 4: serve mode — job protocol, worker pool, in-memory temp store #5354) should hold oneAtlasEmbedderper process (or a small pool behind the same mutex shape); session load is the ~1.6 GB RSS part and is shared state by design. Never an embedder per request. The CLI (Atlas step 3: CLI mode — one-shot commands at Python surface parity #5353) constructs one per process run.run_search(&dyn SearchBackend, &dyn QueryEmbedder, &Filters, &SearchOptions) -> Result<serde_json::Value>andrun_get_chunk(...);Filters::new(...)carries the Python's validation/fatal texts;SearchOptions::default()= production defaults (hybrid, k 10, pool 100, ceiling lives inFilters, defaultpersonal).libstdc++.so.6+libz.so.1resolvable from the closure (hit with the v3 venv's libonnxruntime during the smoke; noted in the example's docs).Closing — the step gate fully passes. Next per the epic chain: step 3 (#5353, CLI mode).