design(atlas): retrieval statistics, result feedback, and query traces — top-K survival, thumbs + essays, per-stage scores #6027

Activity

1. dezren39

   added

   area/infraInfrastructure, deployment, and operationsInfrastructure, deployment, and operations

   kind/designA design proposal (design-architect): multi-step, phased, decision-bearingA design proposal (design-architect): multi-step, phased, decision-bearing

   on Nov 5, 2026
2. developing-today commented on Nov 5, 2026

   

   developing-todayBot

   on Nov 5, 2026 – with developing.today

   Contributor

   More actions

   CODE-1693
3. dezren39 commented on Nov 5, 2026

   

   dezren39

   on Nov 5, 2026

   MemberAuthor

   More actions

   Research hard constraint: traces are logged AT QUERY TIME, or they don't exist

   Cluster 4's one hard constraint, which this design must treat as load-bearing:

   • Per-stage scores cannot be recomputed after the corpus moves. Fan-out → merge → rescore scores depend on the exact corpus state and embed versions at query time. Any design that doesn't log at query time has no analytics — retrofitting is impossible by construction. The trace record per query: query_id, timestamp, querier context, raw query + rewritten query (the dual-submission rewriter makes these differ), corpus snapshot ids (per-shard head markers), and per candidate: embed id + embed version id, shard, per-stage (score, rank) at fan-out/merge/rescore, final rank, and which K cutoffs it survived. Ranks must be logged for the full candidate list at each stage, not just final top-K — survival at K=3 is only computable if stage outputs were recorded deeper than 3. Top-K survival stats (K=1/2/3/5 by shard, tier, source type) are then funnel aggregation over this log, not a separate system.
   • Feedback binding is (query_id, result_id, embed_version_id) — feedback against a version, or retention/versioning changes make old feedback uninterpretable.
   • Thumbs are sparse and negatively biased — pair them with implicit signals (copy ≈ success, requery/regenerate ≈ failure, dwell) from day one; neither raw thumbs nor implicit counts are auto-reranking inputs at personal scale (position bias). Their highest-value use: negative-feedback queries are promoted into a versioned golden set with corrected ground truth, and the golden set gates chunking/rescore/model changes. Free-text essays normalize via an LLM judge with an explicit rubric (labels like wrong-source / stale-version / right-doc-wrong-span / missing-doc / ranking-only), calibrated by periodic hand-scored samples — an uncalibrated judge drifts silently.

   Wave-2 children filed under this issue

   • #6046 — feedback-derived eval set gating index changes
   • #6047 — clicks/copies/dwell/requery as weak labels

   — Wave-2 research fold-in, 2026-11-04 (prior-art doc: ~/workspace/scratch/atlas-ingestion-brief-2026-11-04/prior-art-research.md)
4. dezren39

   added

   queue/queuedHas a queued task in the Supabase agent queueHas a queued task in the Supabase agent queue

   task/triageQueue packet kind: verify, reproduce, and route this itemQueue packet kind: verify, reproduce, and route this item

   on Nov 5, 2026
5. dezren39 commented on Nov 5, 2026

   

   dezren39

   on Nov 5, 2026

   MemberAuthor

   More actions

   🤖 Agent note · handed off

   • Agent: queue worker (repo-worker-code), lane code-v
   • Session / ID: task 10394
   • Found via: queue claim
   • Task: Triage issue design(atlas): retrieval statistics, result feedback, and query traces — top-K survival, thumbs + essays, per-stage scores #6027 — assigned to the opencode execution lane; handed off to the laptop claimant, not worked by this lane.
   • Expectation: claimant launches the bound opencode session for task 10394.
   • Links: n/a
6. dezren39

   added

   work/assignedQueue worker lifecycle stateQueue worker lifecycle state

   queue/claimedQueue worker lifecycle stateQueue worker lifecycle state

   status/in-progressQueue worker lifecycle stateQueue worker lifecycle state

   and removed

   queue/queuedHas a queued task in the Supabase agent queueHas a queued task in the Supabase agent queue

   on Nov 5, 2026
7. github-actions

   assigned

   dezren39

   on Nov 5, 2026
8. dezren39 commented on Nov 5, 2026

   

   dezren39

   on Nov 5, 2026

   MemberAuthor

   More actions

   🤖 Agent note · handed off

   • Agent: queue worker (repo-worker-code), lane code-e
   • Session / ID: task 10394
   • Found via: queue claim
   • Task: handing off task 10394 for issue design(atlas): retrieval statistics, result feedback, and query traces — top-K survival, thumbs + essays, per-stage scores #6027 (design(atlas): retrieval statistics, result feedback, and query traces) to the opencode claimant (assigned_lane=opencode); claimant launches the bound session.
   • Expectation: bound opencode session drives task 10394 to a terminal state; this lane does not work it directly.
   • Links: issue design(atlas): retrieval statistics, result feedback, and query traces — top-K survival, thumbs + essays, per-stage scores #6027
9. dezren39 commented on Nov 5, 2026

   

   dezren39

   on Nov 5, 2026

   MemberAuthor

   More actions

   Agent note - handed off

   • Agent: opencode claimant (opencode-claimant)
   • Session / ID: queue task 10394; bound session ses_e5e5390ebb4ffeBNW4a72aN77s
   • Found via: queue task handoff on the opencode execution lane
   • Task: Triage design(atlas): retrieval statistics, result feedback, and query traces — top-K survival, thumbs + essays, per-stage scores #6027: design(atlas): retrieval statistics, result feedback, and query traces — top-K survival, t
   • Expectation: the bound worker session works task 10394 to a terminal queue state and must not claim another task. It revalidates the issue/PR before acting.
   • Links: design(atlas): retrieval statistics, result feedback, and query traces — top-K survival, thumbs + essays, per-stage scores #6027
10. dezren39 commented on Nov 5, 2026

    

    dezren39

    on Nov 5, 2026

    MemberAuthor

    More actions

    🤖 Agent note · handed off

    • Agent: queue worker (repo-worker-code), lane code-v
    • Session / ID: task 10394
    • Found via: queue claim
    • Task: Triage of issue design(atlas): retrieval statistics, result feedback, and query traces — top-K survival, thumbs + essays, per-stage scores #6027 — execution lane assigned opencode; handed off to the opencode claimant, which launches the bound session.
    • Expectation: claimant session drives task 10394 to a terminal state under the queue worker id.
    • Links: issue design(atlas): retrieval statistics, result feedback, and query traces — top-K survival, thumbs + essays, per-stage scores #6027
11. 48 remaining items

    Load more
12. dezren39

    added

    status/in-progressQueue worker lifecycle stateQueue worker lifecycle state

    on Nov 6, 2026
13. dezren39 commented on Nov 6, 2026

    

    dezren39

    on Nov 6, 2026

    MemberAuthor

    More actions

    🤖 Agent note · handed off

    • Agent: queue worker (repo-worker-code), lane code-r
    • Session / ID: task 10394
    • Found via: queue claim
    • Task: Handing task 10394 for issue design(atlas): retrieval statistics, result feedback, and query traces — top-K survival, thumbs + essays, per-stage scores #6027 to the opencode claimant (execution lane assigned opencode); the bound opencode session will work issue design(atlas): retrieval statistics, result feedback, and query traces — top-K survival, thumbs + essays, per-stage scores #6027 to its terminal state.
    • Expectation: claimant launches a bound session; task 10394 parked 1800s awaiting launch.
    • Links: issue design(atlas): retrieval statistics, result feedback, and query traces — top-K survival, thumbs + essays, per-stage scores #6027
14. dezren39

    removed

    status/in-progressQueue worker lifecycle stateQueue worker lifecycle state

    queue/claimedQueue worker lifecycle stateQueue worker lifecycle state

    on Nov 6, 2026
15. dezren39 commented on Nov 6, 2026

    

    dezren39

    on Nov 6, 2026

    MemberAuthor

    More actions

    🤖 Agent note · handed off

    • Agent: queue worker (repo-worker-code), lane code-q
    • Session / ID: task 10394
    • Found via: queue claim
    • Task: handing task 10394 (issue design(atlas): retrieval statistics, result feedback, and query traces — top-K survival, thumbs + essays, per-stage scores #6027) to the opencode claimant per execution-lane assignment; claimant launches the bound opencode session
    • Expectation: done when the work in issue design(atlas): retrieval statistics, result feedback, and query traces — top-K survival, thumbs + essays, per-stage scores #6027 is implemented and verified by the claimant session
    • Links: issue design(atlas): retrieval statistics, result feedback, and query traces — top-K survival, thumbs + essays, per-stage scores #6027
16. dezren39 commented on Nov 6, 2026

    

    dezren39

    on Nov 6, 2026

    MemberAuthor

    More actions

    🤖 Agent note · handed off

    • Agent: repo queue worker, lane code-a (repo-worker-code)
    • Session / ID: cron repo-queue-worker-shift run 2026-11-05 22:40 CST; task 10394
    • Found via: claimed task 10394 this shift; execution-lane assignment returned opencode
    • Task: task 10394 for issue design(atlas): retrieval statistics, result feedback, and query traces — top-K survival, thumbs + essays, per-stage scores #6027 (triage). Fresh check via REST: issue design(atlas): retrieval statistics, result feedback, and query traces — top-K survival, thumbs + essays, per-stage scores #6027 is OPEN, labels task/triage, status/parked, queue/parked, area/infra, work/assigned, kind/design, no hard-stop labels (held / user/owned / do-not-merge / do-not-auto-edit). Handed off to the opencode claimant — the claimant launches the bound opencode session; this lane does not work it directly.
    • Expectation: claimant session drives task 10394 to a terminal state. Attempts 8/12.
    • Links: task 10394 · issue design(atlas): retrieval statistics, result feedback, and query traces — top-K survival, thumbs + essays, per-stage scores #6027
17. dezren39

    added

    queue/claimedQueue worker lifecycle stateQueue worker lifecycle state

    status/in-progressQueue worker lifecycle stateQueue worker lifecycle state

    and removed

    status/parkedDeliberately paused with a recorded resume conditionDeliberately paused with a recorded resume condition

    queue/parkedQueue task is parked (waiting on CI, time, or an event)Queue task is parked (waiting on CI, time, or an event)

    on Nov 6, 2026
18. dezren39 commented on Nov 6, 2026

    

    dezren39

    on Nov 6, 2026

    MemberAuthor

    More actions

    🤖 Agent note · picked up

    • Agent: repo queue worker (repo-worker-code), OpenChamber scheduled lane 7b47536b-c162-444c-be8a-3a9538cc4cb5
    • Session / ID: task 10394
    • Found via: atomic claim from repo-health-code-pilot
    • Task: Triage issue design(atlas): retrieval statistics, result feedback, and query traces — top-K survival, thumbs + essays, per-stage scores #6027 — retrieval statistics, result feedback, and query traces.
    • Fresh check: issue is OPEN; no hard-stop labels; no active PR found by issue-number search. I am revalidating related design work before routing.
    • Next: post evidence-backed triage and route any remaining design follow-up.
19. dezren39 commented on Nov 6, 2026

    

    dezren39

    on Nov 6, 2026

    MemberAuthor

    More actions

    🤖 Agent note · triaged

    • Agent: repo queue worker (repo-worker-code), OpenChamber scheduled lane 7b47536b-c162-444c-be8a-3a9538cc4cb5
    • Session / ID: task 10394
    • Fresh state: design(atlas): retrieval statistics, result feedback, and query traces — top-K survival, thumbs + essays, per-stage scores #6027 is OPEN, has no hard-stop labels, and has no open PR tied to it. Parent atlas ingestion program: semi-automated updates, versioning, provenance, feedback — long-term design + tasks (2026-11-04 brief) #6018 is OPEN; children design(atlas): golden-set builder + eval gate — feedback-derived versioned eval set gating index changes #6046 and design(atlas): implicit-signal logging — clicks, copies, dwell, requery as weak feedback labels #6047 are OPEN. No superseding work found.
    • Current-main check: origin/main at 32ec450e749cc40b2904682ffd6fd968e6081d38. The existing pkgs/atlas-search/crates/atlas-core/src/eval.rs / eval/README.md harness evaluates saved rankings offline; its run contract is query ID → ranked document IDs and says raw scores are not required. search.rs / fusion.rs hold branch ranks and fused scores in the live search path, but the inspected path has no durable per-query trace or feedback capture. This is not already implemented or superseded.

    Triage / route

    • Keep design(atlas): retrieval statistics, result feedback, and query traces — top-K survival, thumbs + essays, per-stage scores #6027 as the parent design; no code fix or duplicate follow-up task should be opened. Existing children already partition the adjacent design work: design(atlas): golden-set builder + eval gate — feedback-derived versioned eval set gating index changes #6046 (feedback-derived golden set/eval gate) and design(atlas): implicit-signal logging — clicks, copies, dwell, requery as weak feedback labels #6047 (implicit signals).
    • Design ordering: first specify query-time trace capture and retention, since stage scores and corpus/embed state cannot be reconstructed after the corpus changes. Capture the raw/rewritten query, shard snapshot, and the full candidate list's per-stage score/rank (not only top-K), with embed/version identity. Then derive per-shard/per-embed top-K survival and post-merge top-1/2/3/5 distributions; bind feedback to (query_id, result_id, embed_version_id); specify feedback surfaces, essay normalization, bias/anti-gaming, and which product/retention/ranking decisions each signal may feed.
    • Dependencies: align query parameters with open atlas search: query-time parameter surface — every retrieval knob as a CLI flag / raw-query field, no hardcoded constants (extends #5741, #5668) #5774 and rescore behavior with open Atlas search: Jaccard as a third signal (rescore + ingest dedup); SPLADE as a triggered follow-on #5733. Atlas search: SPLADE investigation — third retrieval signal, eval harness, query-adaptive weighting #5741 is CLOSED; its current-main eval harness is useful as an offline consumer, not a substitute for query-time traces.
    • Blockers: none to continue the design. The load-bearing implementation constraint is query-time logging; no runtime defect was alleged, so reproduction is not applicable. No external decision is needed to draft the design; record any unresolved product choices in design(atlas): retrieval statistics, result feedback, and query traces — top-K survival, thumbs + essays, per-stage scores #6027 rather than blocking this triage.

    No follow-up queue task was added because the parent design and the two relevant children already exist.
20. dezren39

    removed

    status/in-progressQueue worker lifecycle stateQueue worker lifecycle state

    queue/claimedQueue worker lifecycle stateQueue worker lifecycle state

    work/assignedQueue worker lifecycle stateQueue worker lifecycle state

    on Nov 6, 2026