Observability: base LLM tracing service (Arize Phoenix) with prompt/response inspection #4

Activity

1. prometheusgr

   added

   enhancementNew feature or requestNew feature or request

   on Aug 31, 2026
2. prometheusgr commented on Aug 31, 2026

   

   prometheusgr

   on Aug 31, 2026

   OwnerAuthor

   More actions

   Breaking this into slices per the project's thin-vertical-slice discipline (see AGENTS.md Definition of Done — "no unresolved scope creep").

   This issue (#4) is now scoped to just the base tracing slice:

   • Stand up Arize Phoenix as a self-hosted docker-compose service (and later a K8s manifest in Phase 5, same pattern as doc-search)
   • Instrument the LangGraph chat agent (OpenInference auto-instrumentation over OTel) so every chat turn is captured as a trace tree: user input → LLM call(s) → tool call(s) → final output, with latency and token counts per span
   • Prompt/response inspection: click into a trace in the Phoenix UI and see the exact prompt sent to llama-server, the response, and which tools fired
   • docs/OBSERVABILITY.md covering what's captured, how to view it, and air-gap vendoring notes for the Phoenix image

   Split into separate follow-up issues (each depends on this slice landing first):

   • Observability: trace clustering/search over historical traces #29 — trace clustering/search over historical traces (the tool-mining use case)
   • Observability: evaluation hooks (LLM-as-judge, human annotation, custom scorers) #30 — evaluation hooks (LLM-as-judge / human annotation / custom scorers) — needs its own judge-model config design, not just more instrumentation
   • Observability: cost-per-span tracking #31 — cost-per-span tracking — no cost model exists yet for the default self-hosted llama.cpp path; deferred rather than shipping an always-zero column

   Cost-per-span from the original bullet list is covered by #31, not this issue.
3. prometheusgr

   changed the title [-]Add an LLM Observability Sevice[/-] [+]Observability: base LLM tracing service (Arize Phoenix) with prompt/response inspection[/+] on Aug 31, 2026
4. prometheusgr commented on Aug 31, 2026

   

   prometheusgr

   on Aug 31, 2026

   OwnerAuthor

   More actions

   One more split-off, caught during planning: #32 — configurable trace retention (session-only / N days / forever, same pattern as the existing conversation/knowledge-base/API-key retention windows). This repo has a hard expectation that every persisted store gets a retention policy (Phase 4b) — traces are no exception, but they live in Phoenix's own storage (not the eaistack Postgres DB this repo's retention sweep owns), so purging them needs a different mechanism than the existing sweep's direct DB deletes. Scoped separately rather than blocking this slice on verifying Phoenix's deletion API, but tracked to land immediately after #4, ahead of #29/#30/#31, since trace data starts accumulating the moment #4 merges.
5. prometheusgr

   added a commit that references this issue on Aug 31, 2026

   Merge pull request #34 from prometheusgr/feature/issue-4-llm-observab…

   ...

   9df1fb9
6. prometheusgr

   closed this as completedon Aug 31, 2026