design(atlas): golden-set builder + eval gate — feedback-derived versioned eval set gating index changes #6046

Activity

1. dezren39

   added

   area/infraInfrastructure, deployment, and operationsInfrastructure, deployment, and operations

   kind/designA design proposal (design-architect): multi-step, phased, decision-bearingA design proposal (design-architect): multi-step, phased, decision-bearing

   on Nov 5, 2026
2. developing-today commented on Nov 5, 2026

   

   developing-todayBot

   on Nov 5, 2026 – with developing.today

   Contributor

   More actions

   CODE-1711
3. dezren39

   added

   queue/queuedHas a queued task in the Supabase agent queueHas a queued task in the Supabase agent queue

   task/triageQueue packet kind: verify, reproduce, and route this itemQueue packet kind: verify, reproduce, and route this item

   work/assignedQueue worker lifecycle stateQueue worker lifecycle state

   queue/claimedQueue worker lifecycle stateQueue worker lifecycle state

   status/in-progressQueue worker lifecycle stateQueue worker lifecycle state

   and removed

   queue/queuedHas a queued task in the Supabase agent queueHas a queued task in the Supabase agent queue

   on Nov 5, 2026
4. dezren39 commented on Nov 5, 2026

   

   dezren39

   on Nov 5, 2026

   MemberAuthor

   More actions

   🤖 Agent note · done (triage)

   • Agent: queue worker (repo-worker-code), lane code-k
   • Session / ID: task 10412
   • Found via: queue claim
   • Task: Triage issue design(atlas): golden-set builder + eval gate — feedback-derived versioned eval set gating index changes #6046 (design(atlas): golden-set builder + eval gate)
   • Expectation: design ticket; triage done = scope, inputs, ordering and blockers recorded below
   • Links: parent issue design(atlas): retrieval statistics, result feedback, and query traces — top-K survival, thumbs + essays, per-stage scores #6027 (OPEN); related issue Atlas search: SPLADE investigation — third retrieval signal, eval harness, query-adaptive weighting #5741, issue atlas search: query-time parameter surface — every retrieval knob as a CLI flag / raw-query field, no hardcoded constants (extends #5741, #5668) #5774 (both OPEN)

   Triage findings (verified fresh 2026-11-05 via gh + local source check):

   • Issue design(atlas): golden-set builder + eval gate — feedback-derived versioned eval set gating index changes #6046 is OPEN. Labels: task/triage, queue/queued, area/infra, kind/design. No hard-stop labels (held, user/owned, do-not-merge, do-not-auto-edit) and no user/review / user/escalated. Only comment is an automated Kaneo cross-link. The research source named in the body (~/workspace/scratch/atlas-ingestion-brief-2026-11-04/prior-art-research.md) is present on the sandbox today.
   • What the design must deliver ("done looks like" in the body): a builder + gate design — golden-set schema, the promotion flow from feedback, and a gate contract per change class.
   • Scope split (from the body): (1) Builder — thumbs-down + sampled queries become candidate eval entries with corrected expected results, a human/agent confirmation step, and versioning of the set itself (target size per the research: 50–200 real queries). (2) Eval gate — which change classes must run it (chunking policy, rescore weights, embedding model, fan-out config), which metrics (RAGAS-style / RAG triad named in the research), and what delta blocks a change. (3) Maintenance — retire/repair rules for stale entries so the gate does not fossilize as the corpus moves.
   • Key distinction from the sibling harness: issue Atlas search: SPLADE investigation — third retrieval signal, eval harness, query-adaptive weighting #5741 (SPLADE eval harness, handed off to the opencode claimant earlier today as task 10153) is signal-specific; this ticket is the corpus-wide gate. The two must share the golden set rather than grow separate eval corpora — the design here owns the set and the gate; issue Atlas search: SPLADE investigation — third retrieval signal, eval harness, query-adaptive weighting #5741's harness is one consumer.
   • Ordering / dependencies: the builder's fuel is the feedback + query-trace capture designed in the parent, issue design(atlas): retrieval statistics, result feedback, and query traces — top-K survival, thumbs + essays, per-stage scores #6027 (OPEN) — the promotion flow cannot be finalized before the parent settles what feedback/traces are recorded, though the set schema and gate contract can be drafted against it now with assumptions stated. One named gated change class (rescore weights) is the parameter surface of issue atlas search: query-time parameter surface — every retrieval knob as a CLI flag / raw-query field, no hardcoded constants (extends #5741, #5668) #5774 (OPEN); the gate contract should reference that surface rather than redefine the knobs.
   • Oracle note: the golden set is this system's test oracle — per the repo testing doctrine, the design should name it as such (versioned set + threshold deltas as the decision procedure for index changes), which is exactly what turns chunking/weight/SPLADE experiments into measurable questions.
   • Blockers: none external, no Drew decision needed to draft. No implementation is scoped by this triage.

   Triage complete for task 10412; issue #6046 stays open for the builder + gate design.
5. dezren39

   removed

   status/in-progressQueue worker lifecycle stateQueue worker lifecycle state

   queue/claimedQueue worker lifecycle stateQueue worker lifecycle state

   work/assignedQueue worker lifecycle stateQueue worker lifecycle state

   on Nov 5, 2026
6. github-actions

   assigned

   dezren39

   on Nov 5, 2026