Repository navigation
Atlas search: Jaccard as a third signal (rescore + ingest dedup); SPLADE as a triggered follow-on #5733
Description
Atlas search: Jaccard as a third signal (rescore + ingest dedup); SPLADE as a triggered follow-on
Context: the atlas hybrid design settled 2026-11-04 (see the shard-restore work under #5669 and the spike verdict on hatch-personal#20): retrieval = dense cosine scan + FTS/BM25, one fan-out per shard carrying both scorers, merged with reciprocal rank fusion. Dot product adds no independent signal on normalized embeddings. Drew (2026-11-04) asked where Jaccard fits in atlas specifically — this issue records the answer as a design task.
Proposal: Jaccard as a THIRD signal, at rescore — not at retrieval
- Each chunk gains a stored token-shingle set (hashed word shingles) alongside its embedding and text.
- Retrieval stays two-scorer (cosine + BM25). Jaccard is computed only over the candidate union (the top ~100 merged candidates), as a rescore-stage feature:
- Near-exact boost: a candidate whose shingle overlap with the query is very high is probably the text the user pasted or quoted (an error message, a passage, a title) — cosine alone ranks it "topically similar among many"; Jaccard identifies it as the match. Boost it decisively.
- Tie-breaker / confirmation: among semantically-equal candidates, prefer higher literal overlap when the query contains rare exact tokens.
- Why not a full-corpus third scorer: set intersection per row across every shard is real scan cost for a signal that only matters near the top. Rescore-stage keeps it ~free.
Second use: ingest / promotion dedup
- At ingest (and at staging→core promotion, atlas staging tier: versioned schema, registry, aggregator with merge-on-close promotion into three-level core #5669), compute shingle overlap of a new chunk against recent / same-source chunks: ≥ ~0.9 Jaccard = near-duplicate — flag or suppress instead of silently double-indexing. The corpora most at risk: brain-dump explosions, repeatedly-versioned design docs, re-ingested repos.
- Dedup findings are data, not deletions: record the edge (chunk, near-dup-of, score) and let promotion policy decide; never auto-delete on a similarity score alone.
SPLADE — investigated only on a trigger (Drew, 2026-11-04: "no reason to investigate SPLADE?")
There IS a reason — quality, not speed: learned sparse retrieval can beat BM25 where vocabulary mismatch hurts (the model expands terms with learned weights), and it would collapse hybrid retrieval into a single vector scan (TursoDB even has an experimental sparse-IVF index). The costs are also real: a SPLADE encoder must run at ingest and at query time, a model must be hosted, and our corpora are technical text where exact terms dominate — BM25's home turf.
So: triggered follow-on, not a standing task. Investigate SPLADE only if the shipped hybrid's held-out evaluation shows the lexical half is the weak side (e.g. recall@10 on the eval set improves materially when an oracle lexical ranker is substituted). Bar for adoption: a measured recall@10 gain over the BM25 half on the held-out query set, large enough to justify a permanent inference dependency at ingest + query time. Until the trigger fires, this stays a named paragraph, not a project.
Acceptance shape
- Design note covering: shingle construction (size, hashing, storage cost per chunk), the rescore formula (how the Jaccard feature enters the fused ranking), dedup thresholds + the promotion-policy hook, and the eval-set methodology that would (a) prove the rescore signal helps and (b) arm the SPLADE trigger.
Activity
developing-today commentedon Nov 5, 2026
CODE-1423
dezren39 commentedon Nov 5, 2026
Update (2026-11-04): the SPLADE follow-on below has been promoted to its own issue at Drews direction — see the SPLADE investigation issue (eval harness + query-adaptive weighting). The trigger framing there replaces the paragraph in this issue: ingest cost is acceptable per Drew, the bar is measured quality gain at query time in the current latency band.
dezren39 commentedon Nov 5, 2026
Wave-2 research addition: the ingest-time companion pipeline (MinHash/LSH → verified Jaccard edges)
This issue's Jaccard work is rescore-time (query-side). The Atlas prior-art pass (Cluster 3) specifies the ingest-time companion — same math, different table — so the two don't grow into rival implementations:
(chunk_a, chunk_b, jaccard, method, run_id); cluster by connected components, one canonical member per cluster, discarded→canonical map.Two discipline notes from the research:
SimHash is the alternative worth one comparison experiment for very short texts (tiny shingle sets); not a commitment. The edges recorded here feed lineage (#6026) and the staging gate's dedup stage (#6057).
Filed under: Atlas ingestion program #6018, wave-2 research fold-in (research item 18).
— Wave-2 research fold-in, 2026-11-04 (prior-art doc:
~/workspace/scratch/atlas-ingestion-brief-2026-11-04/prior-art-research.md)dezren39 commentedon Nov 5, 2026
🤖 Agent note · triaged
Verified fresh, 2026-11-05:
What remains (the whole remaining scope): the design note this issue's Acceptance shape names — (1) shingle construction (size, hashing, storage cost per chunk) + the shared normalizer, (2) the rescore formula for how Jaccard enters the fused cosine+BM25 ranking over the ~100-candidate union (near-exact boost, rare-token tie-break), (3) dedup thresholds + promotion-policy hook settled jointly with #6057/#6026 so there is one implementation, and (4) the held-out eval methodology that proves the rescore signal helps. Enqueued as task 10487.
Ordering: the note's dedup/edge sections should be cross-checked against the current #6057 and #6026 texts before posting (they are the live consumers); its eval methodology should reuse the eval-harness work in #5741/#5669 rather than inventing a second one. No blockers. This issue stays open until task 10487's note lands.
developing-today commentedon Nov 5, 2026
🤖 Agent note · done
Design note — Jaccard as third signal (rescore) + ingest dedup (task 10487)
1. Shingle construction, storage, and the shared normalizer
atlas-normalizefunction used by BOTH the query-side rescore path and the ingest dedup path — there is exactly one implementation. Steps, in order: Unicode NFKC → lowercase → strip boilerplate (fenced-code markers, repeated headers/footers, session chrome) → collapse whitespace → tokenize on non-alphanumeric boundaries (code identifiers kept whole). Versioning rule: every stored shingle set, MinHash signature, and dedup edge carriesnormalizer_version(integer, starts at 1). A normalizer change bumps the version; sets/edges of different versions are never compared — affected chunks are re-shingled lazily on next ingest/touch, and dedup runs group by version. This matches the wave-2 fold-in on this issue (normalizer version is part of the edge record).u64array.2. Rescore formula (exact)
Retrieval is unchanged: dense cosine + BM25 per shard, reciprocal rank fusion, candidate union = top 100 merged candidates.
For each candidate
cwith queryq:J(c) = |S_q ∩ S_c| / |S_q ∪ S_c|over the shingle sets (samenormalizer_versiononly; a version-mismatched candidate getsJ = 0and is flagged for re-shingling).Let
rrf_norm(c)= the candidate's RRF score divided by the union's max RRF score (so the top retrieval candidate is 1.0). Then:J(c) ≥ 0.60,final(c) = 1.0 + J(c)— this tier always outranks every non-near-exact candidate (their max is 1.0, below). Among near-exact candidates, higher J wins. Rationale: ≥0.6 shingle overlap on 5-shingles means the candidate is, with very high confidence, the pasted/quoted text itself.final(c) = 0.85 × rrf_norm(c) + 0.15 × J(c).finalscores differ by < 0.005 are ordered byJdescending, then byrrf_normdescending, then by chunk id ascending (deterministic).3. Dedup thresholds and edge schema (consistent with issue #6057 / issue #6026)
Ingest pipeline (the #6057 staging gate's dedup stage runs exactly this; there is one implementation, not one per consumer):
Thresholds:
J ≥ 0.90→ a dedup edge is recorded and the pair joins a near-duplicate cluster (connected components; one canonical member per cluster, chosen as the earliest-ingested, discarded→canonical map kept).0.80 ≤ J < 0.90→ no edge; the pair is counted in run metrics only (below the bar for action, kept visible for threshold tuning).J < 0.80→ nothing.Edge schema (one shape, all consumers):
(chunk_a, chunk_b, jaccard, method, run_id, normalizer_version)wheremethod ∈ {exact_sha256, minhash_verified},chunk_a < chunk_bby id, pluscreated_at. This is the wave-2 edge on this issue, extended only withnormalizer_version(§1's rule).Boundary with issue #6026: dedup edges are NOT lineage edges. #6026's version/rename lineage tracks versions of one object; dedup edges link distinct chunks that happen to be near-identical. Dedup findings are data, never deletions: suppression from search happens via the cluster's canonical-member rule at the #6057 publish step, the underlying chunks stay, and #6026's keep-all rule (#3125) is untouched — nothing is auto-deleted on a similarity score.
4. Eval methodology
α(the 0.15 weight) and the 0.60 near-exact threshold once on a separate dev split; if it still fails, the rescore signal does not ship — the ingest-dedup half (§3) is independent and ships on its own precision spot-check (≥ 95% of sampled ≥0.9 edges confirmed near-duplicate by a human reader).