Repository navigation
Description
Extends #5741 (SPLADE investigation + eval harness) and #5668 (query-embed cache + flags). Design commitments from Drew's direction 2026-11-04.
Principle
Anything not baked into how data is stored must be settable per query — from the fleet-binary CLI flags AND from the raw query format (the full-access path), with identical parameter names in both — so values can be benchmarked and better defaults found without editing code.
If a retrieval constant lives in code today, that's a bug against this principle.
Knob inventory (v1, extensible)
- Per-shard top-k — how many results each shard returns into the fan-out.
- Global fan-in top-k — how many combined results survive the merge.
- Per-signal weights — dense, BM25, SPLADE (the Atlas search: SPLADE investigation — third retrieval signal, eval harness, query-adaptive weighting #5741 adaptive-weighting defaults, overridable per query).
- Per-signal multipliers — score scaling per signal, distinct from the weights.
- Jaccard rescore depth — candidate count entering the Atlas search: Jaccard as a third signal (rescore + ingest dedup); SPLADE as a triggered follow-on #5733 rescore stage.
- Jaccard boost threshold — the overlap at which the rescore boost applies.
- RRF constant k — if reciprocal-rank fusion is used in the merge, its k is a knob, never a hardcoded 0.6.
- Stage toggles — planner and spell correction off / deprioritized / on (companion issue: query-understanding front-end, extends Atlas search: SPLADE investigation — third retrieval signal, eval harness, query-adaptive weighting #5741).
- Hint override — force the query class (code / docs / error / concept) instead of the classifier's call.
- Variant selection — raw / rewritten / both.
- Expand-N — atlas-search service: query-embedding + snapshot result caching, --no-cache/--wait/--expand/--raw flags #5668's existing
--expand Nflag folds into this surface rather than living as a one-off.
Escape hatches as first-class presets
Power users must not need to edit code or config to say "just run my query":
--raw= planner off, spell off, query verbatim, no expansion.- Per-signal weight 0 via flag = signal removed for that query. "No BM25" is one flag, not a config edit — distinct from Atlas search: SPLADE investigation — third retrieval signal, eval harness, query-adaptive weighting #5741's adaptive-weighting rule (weights floored, never zeroed), because that rule binds the automatic classifier, not an explicit user instruction. An explicit 0 is the user's call and is honored.
Defaults are data, not code
- The shipped default vector is whatever Atlas search: SPLADE investigation — third retrieval signal, eval harness, query-adaptive weighting #5741's eval harness last measured best; it ships as a versioned default set, not as constants scattered through the binary.
- Benchmarking = grid runs over the flag space, with the full parameter vector logged alongside the results of every run, so any number is reproducible and attributable to its knobs.
Provenance
Results carry which signals and which variants produced them (the companion issue's raw-only / rewritten-only / both tags, plus per-signal contribution), so a knob change's effect is diagnosable from the output, not guessed.
Boundary (explicit)
- Storage-side parameters are NOT query-time knobs: index tuning (e.g.
max_neighbors), postings construction, shard placement. Changing those is a rebuild/migration decision, governed by docs: Scaler-era Atlas/Turso data architecture revision #5667/atlas staging tier: versioned schema, registry, aggregator with merge-on-close promotion into three-level core #5669, not a flag. - Sensitivity filtering is NOT a knob: it is a safety filter applied before ranking (the id: design: Querying local and remote ID servers v1 #5675/Design: recommendations for ID — similar objects and similar users, across all object types #5732 rules), and no flag may widen it. A parameter surface that can widen visibility is a vulnerability, not a feature.
Open questions
- Preset naming/versioning: how named presets (
--raw, future ones) are versioned so a benchmark citing a preset stays reproducible. - Per-corpus default vectors: the code corpus and the docs corpus may legitimately ship different defaults — one global vector vs per-corpus vectors.
- Guard rails: maximum fan-out caps so an absurd per-shard k can't melt the shards — where the caps live (client, daemon, per-shard) and what error a capped request returns.
Related: #5733 (Jaccard signals), #5667 (Scaler/shard design), #5669 (shard implementation).
Activity
developing-today commentedon Nov 5, 2026
CODE-1460
dezren39 commentedon Nov 5, 2026
Companion issue: #5773 — the query-understanding front-end (spell correction + planner/rewriter) whose stage toggles and variant selection this parameter surface exposes.
dezren39 commentedon Nov 5, 2026
🤖 Agent note · done