Skip to content

Observability: verify and enable TLS termination for the Phoenix Helm chart #33

Open

Description

@prometheusgr

Split from #4 (LLM Observability Service).

Every other Helm chart in this repo (infra/helm/charts/*) defaults tls.enabled: true and terminates TLS itself, using a cert-manager-issued Certificate mounted into the pod. doc-search's chart does this via its own docker-entrypoint.sh, which conditionally adds --ssl-certfile/--ssl-keyfile to its uvicorn invocation — that works because doc-search is a service we build and control the entrypoint of.

Phoenix (infra/helm/charts/phoenix/, added in #4) is a prebuilt, unmodified upstream image (arizephoenix/phoenix) per the "no Dockerfile of our own for prebuilt OSS images" pattern (same as postgres/minio/keycloak). That means we don't control its process invocation the way we do doc-search's, and #4 shipped its chart with tls.enabled: false rather than guess at flags/env vars that might not match the real image's actual TLS support — this repo is air-gapped, so #4's implementation had no way to check Phoenix's docs/CLI against the real running image without fabricating unverified infra.

To resolve:

  1. Verify against the actual vendored arizephoenix/phoenix image whether it supports native TLS termination (env var, CLI flag, or config file) — check its --help output / startup logs / any bundled docs once the image is pulled in a dev environment with internet access.
  2. If it does: wire TLS_ENABLED/cert paths into the chart the same shape as every other service, flip the default to tls.enabled: true, matching the compliance posture every other chart already has.
  3. If it doesn't: add a TLS-terminating sidecar (nginx or envoy) to the Phoenix pod, terminating the cert-manager-issued certificate and proxying plaintext to Phoenix's container port — the fallback pattern for a backing service that can't speak TLS itself. This is real new implementation surface (proxy config to write, test, and maintain), which is why Observability: base LLM tracing service (Arize Phoenix) with prompt/response inspection #4 didn't default to this speculatively.

Until this lands, Phoenix is the one chart in infra/helm/charts/ with tls.enabled: false by default — a documented, deliberate exception (see docs/OBSERVABILITY.md's "Known gap: TLS" section and infra/helm/charts/phoenix/values.yaml's comment), not an oversight.

Activity

  1. added
    type:admin-setupInfra, deployment, ops, or operator/admin setup concern
    effort:MMedium: one layer (backend or frontend), a few days
    area:observabilityTracing, evaluation, cost, Phoenix-related
    and removed
    enhancementNew feature or request
    on Sep 4, 2026
  2. prometheusgr commented on Sep 28, 2026

    @prometheusgr
    OwnerAuthor

    Priority context: the workflow-configuration epic #80 uses Phoenix as the review and diagnosis surface for workflow iteration. Together with #90 (Phoenix authentication), this issue is a prerequisite for relying on Phoenix in that loop, because traces carry user content and the playground can call the model directly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:observabilityTracing, evaluation, cost, Phoenix-relatedeffort:MMedium: one layer (backend or frontend), a few dayspriority:P1High value, near-termtype:admin-setupInfra, deployment, ops, or operator/admin setup concern

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions