Add a lightweight golden-set regression harness for agent/retrieval quality (CI-runnable, non-Phoenix) #41

Context

Flagged during a review of what's missing for a business to safely iterate on this template's chat agent, prompts, and retrieval over time. Issue #36 (Phoenix Datasets/Experiments) and #30 (evaluation hooks: LLM-as-judge, human annotation, custom scorers) both cover observability-integrated evaluation — rich, Phoenix-hosted, and appropriately scoped as bigger follow-on work. Neither covers a much smaller, more immediate need: a lightweight, CI-runnable regression check that answers "did this prompt/retrieval change break something that used to work," without depending on Phoenix, an LLM-as-judge call, or human annotation.

The gap

There is currently no fixture-based regression suite for agent behavior — only unit tests with FakeChatModel (which validate wiring/logic, not answer quality) and the one real-content e2e spec (knowledge-base-search-grounding.spec.ts, requires-profile-llm, excluded from CI per AGENTS.md's e2e section) which checks a single scenario, not a maintained set. A fork that changes chunking, the system prompt, or the retrieval query has no fast, deterministic way to check "does this still retrieve the right document for these ten known questions" before it lands. Today that's manual, one-off testing (as AGENTS.md already asks for retrieval-touching PRs) with no artifact left behind for the next change to reuse.

This is intentionally the "cheap, boring version" of #36/#30 — a golden-set-of-Q&A-pairs-plus-assertions harness, not a judge-model or clustering UI. It can exist independently of whether #36/#30 are ever built, and should be scoped small enough to land without a design decision on judge-model providers (the exact thing #30 is blocked on).

Scope

1. Golden fixture set: a versioned set of question → expected-grounding-document (and optionally expected-answer-substring) pairs, checked into the repo (e.g. backend/tests/eval/golden_set.yaml or similar), seeded against a small fixed knowledge-base fixture — not the live/real knowledge base, so it's deterministic and air-gap-safe.
2. Runner: a script/pytest module that runs the real chat agent (real retrieval against the fixture KB, real LLM per AGENTS.md's --profile llm convention — this cannot be meaningfully faked, same reasoning as the existing grounding e2e spec) against the golden set and reports pass/fail per case (did the expected document get cited in sources, per Phase 4d's structured citations).
3. Non-gating in default CI (same treatment as the --profile llm e2e spec, for the same reason: no GGUF model in the default CI runner) but runnable locally with one documented command, and referenced from AGENTS.md's "Run it before merging..." block alongside the existing grounding e2e spec as another thing to run before merging retrieval/prompt changes.
4. UI element (required): a results view for this harness — at minimum, a "Retrieval Eval" admin screen (require_admin-gated, alongside Settings) that displays the most recent local/manual run's pass/fail results per golden-set case, so a non-technical stakeholder can see eval health without reading a terminal. The run itself stays a local/CI script (per above); the UI's job is only to surface the last recorded result set (e.g. read a committed or uploaded results JSON), not to trigger runs from the browser.

Definition of Done

• Golden fixture set exists, checked into the repo, with at least 8-10 question/expected-source pairs against a fixed test knowledge base.
• Runner script exists, documented in AGENTS.md alongside the existing --profile llm e2e instructions, and produces a machine-readable results artifact (e.g. JSON) consumable by the UI.
• Admin-only "Retrieval Eval" UI screen exists in the frontend, displays the latest results artifact's pass/fail per case, and is reachable from the app's normal navigation.
• Unit tests cover the runner's scoring logic (e.g. "did the expected document ID appear in returned sources") independent of any real LLM call.
• docs/RETRIEVAL_IMPROVEMENT_PROMPTS.md or AGENTS.md updated to reference this harness as part of the retrieval-change verification workflow.
• Standard merge gate from AGENTS.md's Definition of Done applies in full (tests, lint, e2e for the new UI flow, docs).