Observability: Phoenix Datasets/Experiments for prompt regression testing #36

Split from #4 (LLM Observability Service), sibling to #29/#30/#31/#32/#33.

Context

There is currently no regression testing for prompt changes: backend/app/prompts/chat_prompts.py's CHAT_AGENT_SYSTEM_PROMPT is a static string, bumped by hand (version field) with no automated check that a wording change didn't regress answer quality, tool-use reliability, or groundedness. The only signal today is the manual real-model e2e spec (knowledge-base-search-grounding.spec.ts, per AGENTS.md's "run before merging any change touching retrieval" rule) — useful, but it's one hand-run spec, not a repeatable scored regression suite.

Phoenix (self-hosted per #4) ships Datasets + Experiments: curate a set of example inputs (with expected outputs/criteria), run a prompt or the full agent against the dataset, and score + diff results across prompt/model versions over time. Since Phoenix is already ingesting this project's traces, examples can be seeded directly from real captured traces rather than authored from scratch.

This is the direction chosen over adopting promptfoo (a separate CLI-based, provider-agnostic eval framework): Phoenix is already self-hosted, already air-gapped-compatible (#33/#4's constraints already solved), and keeps eval data next to the traces it's evaluating. Adding promptfoo would mean vendoring, air-gapping, and maintaining a second tool with overlapping purpose — redundant with this, not with #30.

Scope

• Build (or seed) a dataset of representative chat-agent inputs, including at least one case that exercises search_knowledge_base tool-calling (the known-fragile path per docs/AGENT_PROMPTS.md/chat_prompts.py's comment on local-model tool-call behavior).
• Wire a repeatable way to run the current CHAT_AGENT_SYSTEM_PROMPT (and future candidate revisions) against that dataset and get a score back — decide whether scoring here reuses Observability: evaluation hooks (LLM-as-judge, human annotation, custom scorers) #30's LLM-as-judge/custom-scorer work or needs only simple deterministic checks (e.g. "did it call the tool," "did it cite a source") to start.
• Decide how a scored experiment run gates a prompt change: informational only (reviewer looks at the diff) vs. a CI-enforced threshold. Given AGENTS.md's TDD posture, lean toward "informational first, enforce later" rather than blocking merges on a nondeterministic LLM score from day one.
• Out of scope: prompt storage/versioning UI (Observability: Phoenix Prompt Hub for prompt storage/versioning and in-UI iteration #35), general trace-quality scoring/annotation (Observability: evaluation hooks (LLM-as-judge, human annotation, custom scorers) #30) — this issue is specifically about dataset-driven regression testing of prompt changes.

Design constraints

• Depends on Observability: evaluation hooks (LLM-as-judge, human annotation, custom scorers) #30's judge-model design decision if LLM-as-judge scoring is used here — don't duplicate that design work, reference it.
• Keep CHAT_AGENT_SYSTEM_PROMPT as the reviewable, git-tracked source of truth (see Observability: Phoenix Prompt Hub for prompt storage/versioning and in-UI iteration #35) — this issue adds a test harness around prompt changes, not a new place prompts live.