Repository navigation
Atlas step 6: golden-parity gate (Gate B) + cutover #5356
Description
Step 6 of the Atlas binary+service epic (#5350). Design of record: doc/2026-11-04T08-07-13Z_design_atlas_binary_service/ (§5 gates B/C + migration phases, §7 Q2/Q3).
Scope
- Gate B run: the golden 311 (golden-queries-v3) through the binary vs the Python production path, both against the production corpus (107,639 chunks; the reference is a same-day Python run on the same corpus, not the historical v3-corpus 189/0.3894). Run through the eval3.py harness shape adapted to call the binary, so per-question rank diffs are inspectable. Tolerances (design §5): hits@10 within ±2 questions, overall MRR within ±0.005, no category (exact-fact / paraphrase / dictated-garbled / cross-session) regressing > 0.01 MRR. Configuration identical both sides, with the Gate C (A/B-winning) vector backend as the default.
- Cutover phases (design §5, additive-first; each flip is Drew's per-surface call):
- CLI opt-in: binary installed as
atlas-searchfrom its own profile position; the Python wrapper renamed — not removed — toatlas-search-py. - Service opt-in: unit enabled on amd; the §3 v1 client tokens issued (with Drew; Q2 decides caps above
personal);--via-servicelive from laptop + sandbox CLIs; standalone remains default. - Default flip per surface (laptop first, sandbox after a soak; Q3 recommends the sandbox default to via-service at this phase).
- CLI opt-in: binary installed as
- Rollback at every phase is repointing the name/flag — there is no data migration anywhere in this program. The Python remains installed as fallback and permanent eval reference after cutover.
Dependencies: step 5; Gate A green (step 1); the A/B verdict (Gate C). Drew's answers to design Q2/Q3 gate phases 2–3 details, not the Gate B run.
Verification gate (exit criteria)
- Gate B results recorded on this issue with the paired per-category table; a tolerance miss stops the cutover (the binary does not flip defaults on a failed gate — the miss gets its own issue and this step stays open).
- After each flip: the deployed
atlas-search --version-equivalent and one golden spot query verified on that surface, and the Python fallback demonstrated still working. - Epic Epic: Atlas as a Rust binary + multi-client search service (b+c) #5350 closes when phase 3 has run on both surfaces or Drew rules a surface stays on the Python.
Out of scope: decommissioning the Python (explicitly never — it is the eval reference), new clients beyond §3 v1, publish-pipeline changes.
Acceptance criteria
- Gate B run completed: golden-311 through the binary vs a same-day Python reference run, both against the production corpus, via the eval3.py-shaped harness; per-category paired table recorded on this issue.
- Tolerances met (design §5): hits@10 within ±2 questions, overall MRR within ±0.005, no category regressing >0.01 MRR — a miss stops the cutover and spawns its own issue.
- Cutover phases (CLI opt-in → service opt-in → per-surface default flip, laptop first) each land only on Drew's per-surface call, with the deployed binary version + a golden spot query verified per surface and the Python fallback demonstrated working.
- Epic Epic: Atlas as a Rust binary + multi-client search service (b+c) #5350 closes when phase 3 has run on both surfaces or Drew rules a surface stays on the Python.
Activity
dezren39 commentedon Nov 4, 2026
Intake review — fleet monitor, 2026-11-04.
Related: epic #5350 (Atlas binary+service); step 5 (#5355, packaging/deploy) and Gate A (step 1) + the Gate C A/B verdict are prerequisites. Design of record:
doc/2026-11-04T08-07-13Z_design_atlas_binary_service/.Notes: the issue's own Verification gate is the substance of the acceptance criteria appended above. The same-day Python reference run on the production corpus is the easy part to get wrong — the historical v3-corpus numbers (189 / 0.3894) are explicitly not the reference.
Risks: phases 2–3 details wait on Drew's answers to design Q2/Q3 — that gates the cutover details, not the Gate B run. The Gate B run itself is unblocked once step 5 lands.
Suggested approach: run Gate B as soon as the step-5 package exists; record the paired table here before any flip conversation.
home: atlas-search serve unit, landed disabled (step 5, #5355)
43 remaining items
dezren39 commentedon Nov 4, 2026
🤖 Agent note · parked
~/atlas/v3/swap-2026-11-03/gateb.logdezren39 commentedon Nov 4, 2026
Step 6 — Gate B verdict + cutover evidence
Gate B: PASS. The service is live and the cutover is complete (one deferred decision, recorded at the end — it was never part of this gate).
Gate B — paired golden-311, unfiltered, scan backend, production corpus
Rust = the packaged binary (step-5 output); Python = the deployed reference CLI, run fresh back-to-back per question (the 05:31 banked Python numbers are shown only for drift context — the fresh pairing is the measurement).
Paired deltas (Rust − Python): Δhits@10 = 0 (gate ±2), ΔMRR = +0.0016 (gate ±0.005). Per-category MRR deltas: exact-fact 0.0000, dictated-garbled 0.0000, paraphrase −0.0032, cross-session +0.0175 — no category regression beyond 0.01 (the only negative is −0.0032; the cross-session delta is an improvement).
Stronger than the bands require: for all 309 questions where both arms returned results, the top-10 lists are byte-identical (309/309). The only two rank disagreements in 311 are one-sided transient Turso errors, and they cancel: v3q058 — the Rust call hit a Turso global timeout at 325.8s (Python ranked the gold 2nd); v3q276 — the Python arm exited SystemExit(2) (Rust ranked the gold 1st). The fresh Python reference also drifted slightly against its own banked run (185→184, −0.0032 MRR) under the concurrent row rewrite, which is why the gate was run paired rather than against the banked numbers.
ORT adjudication: the nix closure's ONNX Runtime 1.27.1 vs the Python pipeline's 1.31.1 produces no measurable ranking drift — consistent with step 1's bit-identity gate. No adjudication action needed.
The filtered pairing was skipped per the step plan (the unfiltered set is the decision set; filtered would have doubled a ~3.5h run). Raw records:
~/atlas/v3/swap-2026-11-03/gateb-results.jsonl(drivergateb_eval.pyalongside). One process death at pair 182 (clean exit, unattributed) was absorbed by the resumable JSONL with zero loss.Default backend, as ruled:
scan. The default is chosen on unfiltered numbers across backends; the index arms have no numbers until the row rewrite completes, so the cutover default is the incumbent scan (banked 185 / 0.3787), and the index-backend decision is explicitly deferred to the A/B completion owned by the rewrite chain (hatch-personal#20).Activation
atlas-search initwas run before any enablement could deploy:~/.local/state/atlas-search/service.json(0600) holds the v1 table as ruled in Q2 — drew-cli cappedsecret-bearing; openchamber / fork-server / sandbox-scripts cappedpersonal.3b3df1be3. It deployed via the fleet switch chain's successful switch (2026-11-04 10:19 CST,SWITCH_EXIT=0, generation/nix/store/w768inbg2z…-nixos-system-amd-26.11.20261102.f9887a8) — two earlier attempts that morning failed inside the chain's own scope (07:04 libvirtd systemd-credential decryption; 09:29 home-manager refusing to clobber the hand-installed broker wants symlinks) and are recorded here only because they gated this step's timing.enabled+active; the running ExecStart is the deployed generation's own build of pkgs/atlas-search (/nix/store/dfs864kg…-atlas-search-0.1.0);/health→{"status":"ok","embedder_state":"ready"}; NRestarts was 0 from first start (09:39, during the chain's activation sequence) until the deliberate battery restart below.tailscale serve --tcp 18760 → 127.0.0.1:18760(tailnet only), i.e.http://amd-6.raptor-pumpkinseed.ts.net:18760. Deviation, recorded: the--httpsform was created first and verified from the laptop, but the sandbox's egress proxy cannot complete TLS to it; the tcp form is the fleet's door pattern and is what the sandbox's tunnel proxy reaches.Client distribution (Q2/Q3 as ruled)
~/.config/atlas-search/client.json= drew-cli. The Python wrapper was renamed to~/.local/bin/atlas-search-py(verified working);atlas-searchnow resolves to the Rust binary via an interim symlink to the step-5 store output. PR home(atlas-search): ship the atlas-search package in home.packages — step 6 cutover (#5356) #5479 (merged49d11b5b7, atlasSearch in home.packages) is not in the current generation — when a later switch deploys it, the profile binary becomes the durable path and the interim symlink should be removed.atlas-search 0.1.0); the model was provisioned into the state dir by copying the laptop's verified copy — both files re-hashed on arrival and matching the pins (tokenizerf1cc44ad…, modelc0c53d7f…;fetch-modelitself is blocked on the sandbox by its proxy's MITM CA, which the binary's rustls stack rightly rejects).~/.config/atlas-search/client.json= sandbox-scripts (personal).~/tools/bin/atlas-searchis now a wrapper that forwards loopback:18760 over the shared laptop-ssh control connection and defaults to via-service — necessary because the via-service client is proxy-blind (follow-up filed: atlas-search: via-service client should honor proxy env (ureq Proxy::try_from_env) #5597); its loud standalone fallback was verified live in exactly that failure mode. The Python wrapper was renamed to~/tools/bin/atlas-search-py(verified working).Verification battery (all against the live service)
max_sensitivity: secret-bearingapplied; sandbox-scripts (requesting secret-bearing) is clamped topersonalin the applied filters and chunk 537 is absent.job_expired_or_unknownaftersystemctl --user restart; health returnsokand the embedder returns toreadyafter a warm query.atlas-search-pyverified working on both machines after the renames.Deferred (not part of this gate)
The index-backend decision (scan vs index-fill vs index-fixed as the default) waits on the row rewrite + the parked index-arm A/Bs — owned by the rewrite chain on hatch-personal#20. When those numbers land, the default flips by config if an index arm wins; nothing in this cutover needs to be redone.
dezren39 commentedon Nov 4, 2026
🤖 Agent note · done