Repository navigation
Epic: Atlas as a Rust binary + multi-client search service (b+c) #5350
Description
Program epic. Drew ruled "b and c" on 2026-11-04 (dezren39/hatch-personal#21): Atlas search ships as (b) a Rust binary vendored from this repo and (c) a multi-client search service built on that binary.
Design of record: doc/2026-11-04T08-07-13Z_design_atlas_binary_service/ (PR #5348, merged 9c5976340). Read it before working any step — the decisions below are summaries, the note is the authority.
Headline decisions
- One Rust binary (
atlas-search,pkgs/atlas-search/), two modes: standalone-capable one-shot CLI +serve. Never thin-client-only. - Service protocol: HTTP+JSON jobs; branch workers concurrent; fused rankings in an in-memory TTL temp store behind per-job capability tokens; fetch-to-fill escalation lives inside the vector branch, paging slices an immutable ranking.
- Auth: per-client bearer tokens in
~/.local/state/atlas-search/service.json, each carrying a maximum sensitivity ceiling the server clamps to. Named clients only; Aster is not a v1 client. Loopback + tailscale serve (MagicDNS), never public. - Vector backend is a pluggable strategy (
scan/index-fill/index-fixed); the golden-311 A/B running on the Python decides the default (design Gate C). - The Python CLI stays the eval reference permanently and the runtime fallback through cutover. Migration is additive-first; there is no data migration (the system is read-only end to end).
Program gates (design §5): Gate A embedder bit-identity on the probe set; Gate B golden-311 tie vs the Python production path on the production corpus (hits@10 ±2, MRR ±0.005, no category regression > 0.01); Gate C A/B winner as implemented default.
Non-goals (design §6): no writes to the production atlas DB (service holds a read-only token only), no corpus re-embedding, no publish-schema changes, no client ceiling widening, no public exposure.
Steps (sub-issues of this epic):
- Embedder parity spike (
atlas-embed) — Gate A - Core search library (
atlas-core) - CLI mode (one-shot commands)
- Serve mode (job protocol, worker pool, temp store)
- Nix packaging + amd deploy
- Golden-parity gate + cutover
Open questions for Drew live in design §7 (Q1 model provisioning, Q2 ceiling caps, Q3 sandbox CLI default flip) — they gate steps 5/6 decisions, not steps 1–4 work.
Acceptance criteria
- Steps 1–6 are attached as native sub-issues of this epic (filed as issues Atlas step 1: embedder parity spike (atlas-embed) — Gate A #5351–Atlas step 6: golden-parity gate (Gate B) + cutover #5356) and their states roll up here.
- Gate A (embedder bit-identity), Gate B (golden-311 tie vs the Python production path) and Gate C (A/B winner as default) results are recorded on the epic or the owning step issue.
atlas-searchbuilds and is packaged via nix (step 5); the Python CLI remains the eval reference and runtime fallback through cutover.- The service binds loopback/tailscale only, enforces per-client sensitivity ceilings server-side, and holds a read-only token only — no writes to the production atlas DB, no corpus re-embedding.
- The cutover decision is recorded with Gate B/C evidence; open questions in design §7 are answered or explicitly deferred before steps 5/6 complete.
Activity
dezren39 commentedon Nov 4, 2026
DESIGN APPROVED + OPEN QUESTIONS RULED (Drew, 2026-11-04 02:31 CST, via the Token usage tracking chat):
Steps 1-4 were already unblocked; steps 5-6 are now unblocked when reached.
dezren39 commentedon Nov 4, 2026
Intake review
Related: PR #5348 (design of record, merged); PR #5367 (step 1, issue #5351 — already open, the program is moving); issues #5351–#5356 (the six steps).
Notes: Program epic ruled by Drew today ("b and c", hatch-personal#21). Filed with no labels at all — normalized in this pass. Still live: step 1 has an open PR; steps 2–6 have no PRs yet.
Risks: Gate B is the hard gate — a golden-311 regression blocks cutover regardless of build progress. Per-client ceiling enforcement is a security property, not a feature: it needs a test, not just code review.
Suggested approach: Keep the epic as the rollup only; work flows through the step issues. First monitor follow-up: confirm the steps are attached as native sub-issues (protocol §11) — they were filed as a set today, so the attach should be verified, not assumed.
🤖 Agent note · filed
11 remaining items
dezren39 commentedon Nov 4, 2026
agent-note:v1
{"kind":"pickup","task":9478,"lane":"c29174f1-48e0-4b13-b4ff-9b73ab67e9ab","execution_lane":"opencode","state":"in-progress","note":"Claimed for fresh triage. Revalidating epic, six step issues, merged/open PRs, and current-main implementation state."}
dezren39 commentedon Nov 4, 2026
Triage outcome — task 9478 (2026-11-04 14:48Z)
Verdict: fix path is scoped; no new epic-level implementation task is needed. The approved design of record is merged as #5348, and
pkgs/atlas-search/is present on fetchedorigin/mainat329b23bb2. All six planned step issues exist. Steps 1–5 (#5351–#5355) are closed; their implementation PRs #5367, #5372, #5377, #5392, and #5439 are merged.Step 6 / Gate B (#5356) remains OPEN and parked, so this epic correctly stays open. No open PR for #5356 was found. Its Gate B paired eval is already running on the laptop under the existing owner/task 9494 (
~/atlas/v3/swap-2026-11-03/gateb_eval.py, PID 782946). At 14:44Z the results file had 235 lines and the log had progressed through v3q118. Do not start a duplicate run.Next order: let task 9494 finish and record its paired per-category Gate B table on #5356; evaluate the design tolerances (hits@10 ±2, MRR ±0.005, no category regression >0.01); only then make the per-surface cutover decisions, with the design’s Drew-owned phase decisions respected. The in-flight Gate B run is tracked by #5356/task 9494, not a blocker requiring a new epic task.
agent-note:v1
{"kind":"done","task":9478,"lane":"c29174f1-48e0-4b13-b4ff-9b73ab67e9ab","execution_lane":"opencode","state":"done","note":"Fresh triage complete: steps 1–5 merged/closed; step 6 Gate B remains open and is actively owned by task 9494. Epic stays open; no duplicate work or new implementation task."}
dezren39 commentedon Nov 4, 2026
CI clarification for the triage record: PR #5367's historical
basic: production line limit (2500)failure was unrelated to Atlas; the failing file waspkgs/id/src/world/core.rsat 2515 lines and the check referenced the existing split issue #1411. That defect was subsequently fixed by PR #5365 / issue #5358; #5358 is now closed and currentorigin/mainhasworld/core.rsat 2194 lines. Gate A itself is recorded as bit-identical in #5351, so this is not a remaining Atlas blocker.dezren39 commentedon Nov 4, 2026
Epic status: complete, except the one deferred decision
Steps 1–6 are all closed. Step 6 (#5356) landed the cutover: Gate B passed with paired deltas of Δhits@10 = 0 and ΔMRR = +0.0016 against the Python reference (309/309 jointly-answered questions byte-identical), the service is enabled and healthy on the laptop (tailnet tcp serve on :18760), clients are distributed per Q2/Q3 (drew-cli secret-bearing on the laptop; sandbox-scripts personal, via-service by default from the sandbox), and the Python CLI lives on as
atlas-search-py, the permanent eval reference. The default vector backend isscan, as ruled.The single piece deliberately left open: the index-backend default decision (scan vs index-fill vs index-fixed). It waits on the Turso row rewrite and the parked index-arm A/Bs, owned by the rewrite chain on hatch-personal#20. When those numbers land, the default flips by configuration if an index arm wins — no step of this program needs to be redone.