Skip to content

Observability: evaluation hooks (LLM-as-judge, human annotation, custom scorers) #30

Open

Description

@prometheusgr

Split from #4 (LLM Observability Service). Depends on the base Phoenix tracing slice landing first.

Score traces to track quality over time, not just latency/cost:

  • LLM-as-judge scorers (needs a design decision: which model judges — the same llama-server instance, a separate judge model/config, or admin-configurable per the existing LLM-provider-config pattern in app.services.system_settings_service)
  • Human annotation (needs a UI surface — Phoenix has one natively; decide whether we expose it as-is or build an in-app equivalent)
  • Custom scorers (deterministic, code-based checks — e.g. "did the answer cite a source when the tool fired")

Needs its own design pass: this is a genuinely separate feature with its own config surface (judge model selection), not just "more instrumentation." Do not bundle into the base tracing PR.

Out of scope for this issue: the base Phoenix service, OTel instrumentation of the chat agent, and trace search/clustering (see the separate issue for that).

Activity

  1. added
    enhancementNew feature or request
    type:featureBrand-new capability that doesn't exist today
    effort:XLExtra large: multi-week, new infra class, or needs its own design pass first
    area:observabilityTracing, evaluation, cost, Phoenix-related
    and removed
    enhancementNew feature or request
    on Aug 31, 2026
  2. prometheusgr commented on Sep 28, 2026

    @prometheusgr
    OwnerAuthor

    Related: #84 (epic #80) introduces the eval-run model (test cases, structural and deterministic content checks, pass rates over N runs, evidence-gated publish). LLM-as-judge was deliberately left out of #84. This issue's judge-model design is where it plugs in, so #30 can build on that substrate rather than defining its own.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:observabilityTracing, evaluation, cost, Phoenix-relatedeffort:XLExtra large: multi-week, new infra class, or needs its own design pass firstpriority:P2Valuable, not urgenttype:featureBrand-new capability that doesn't exist today

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions