You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Observability: evaluation hooks (LLM-as-judge, human annotation, custom scorers) #30
Split from #4 (LLM Observability Service). Depends on the base Phoenix tracing slice landing first.
Score traces to track quality over time, not just latency/cost:
LLM-as-judge scorers (needs a design decision: which model judges — the same llama-server instance, a separate judge model/config, or admin-configurable per the existing LLM-provider-config pattern in app.services.system_settings_service)
Human annotation (needs a UI surface — Phoenix has one natively; decide whether we expose it as-is or build an in-app equivalent)
Custom scorers (deterministic, code-based checks — e.g. "did the answer cite a source when the tool fired")
Needs its own design pass: this is a genuinely separate feature with its own config surface (judge model selection), not just "more instrumentation." Do not bundle into the base tracing PR.
Out of scope for this issue: the base Phoenix service, OTel instrumentation of the chat agent, and trace search/clustering (see the separate issue for that).
Related: #84 (epic #80) introduces the eval-run model (test cases, structural and deterministic content checks, pass rates over N runs, evidence-gated publish). LLM-as-judge was deliberately left out of #84. This issue's judge-model design is where it plugs in, so #30 can build on that substrate rather than defining its own.
Activity
prometheusgr commentedon Sep 28, 2026
Related: #84 (epic #80) introduces the eval-run model (test cases, structural and deterministic content checks, pass rates over N runs, evidence-gated publish). LLM-as-judge was deliberately left out of #84. This issue's judge-model design is where it plugs in, so #30 can build on that substrate rather than defining its own.