Design and implement retry/backoff strategy for outbound service calls #76

Context

docs/CONFIG_AUDIT_2026-09-04.md's solution-wide sweep, done alongside the RAG-configuration audit (#68), confirmed by repo-wide search that no retry/backoff logic exists anywhere in backend/app or mcp-servers/doc-search. Every outbound HTTP call — Keycloak (login/token exchange, JWKS fetch), the LLM (llama-server/OpenAI-compatible), the embedding server, MinIO, and the backend→doc-search MCP call — is a single attempt guarded only by a timeout (see #69 for the timeout-value gaps found in the same sweep). A transient blip (a pod restart, a brief network hiccup, Keycloak momentarily slow) surfaces immediately as a user-facing failure with no retry.

This is a different kind of gap than #68/#69/#70-75: those are "a value exists but isn't configurable." This is "the behavior doesn't exist at all" — it needs its own design pass before implementation, not just a new SystemSettings field.

Why this needs a design pass first, not just an implementation

Several questions need deliberate answers before writing code, per this repo's "decide and document the mechanism" precedent (see how #38 documents the rate-limit store decision, #43 the secrets-rotation decision):

1. Which calls should retry at all? A user-facing synchronous call (chat request → LLM) has different latency tolerance than a background operation (retention sweep, JWKS refresh). Retrying a POST that isn't idempotent (has it already partially succeeded server-side?) needs case-by-case judgment — e.g. is the Keycloak token exchange safe to retry blindly?
2. What backoff policy? Fixed delay, exponential, exponential+jitter — and what bounds (max attempts, max total wait) so a retry loop doesn't silently turn a fast failure into a slow one that blows through the caller's own timeout or the chat rate-limit window (Add rate limiting to chat and auth endpoints #25/Move rate-limit state to a shared store when the backend scales past 1 replica #38).
3. Air-gapped/local-service context matters. Unlike a typical SaaS calling third-party APIs over the public internet, most of these calls are to co-located services on the same cluster (Keycloac, llama-server, embedding-server, doc-search, MinIO) — the failure modes (pod restart, brief unreadiness) are different from "internet flakiness," which should inform whether retry is even valuable versus just failing fast and letting Kubernetes-level restarts/readiness probes handle it.
4. Interaction with existing timeout config (Make auth/MCP client timeouts and JWKS cache settings admin-configurable #69): a retry policy without also considering the per-attempt timeout could multiply worst-case latency unpredictably.
5. Observability: if retries are added, they should be visible somewhere (traces via Observability: base LLM tracing service (Arize Phoenix) with prompt/response inspection #4's Phoenix instrumentation would naturally show retried spans; is that sufficient, or does something need explicit logging?).

Scope (once designed)

1. Document the decision in a new docs/RESILIENCE.md or a new section of docs/ARCHITECTURE.md: which call sites get retry, what policy, what bounds, and why (or why not, for calls deliberately left fail-fast).
2. Implement a small, shared retry helper (not five copy-pasted ad-hoc loops) if multiple call sites need the same policy — but don't build a generic retry framework speculatively if only one or two sites actually need it; follow this repo's no-premature-abstraction standard.
3. Any retry-policy parameters (max attempts, base delay) that are genuinely operator-tunable should follow the established SystemSettings pattern — but don't default to making them configurable just because other things in this repo are; only if the design concludes an operator would plausibly need to tune them per-deployment.
4. Unit tests for the retry logic itself (deterministic — inject a fake clock/sleep per docs/TIME_INJECTION.md rather than real time.sleep in tests).

Non-goals

• This issue is scoped to outbound HTTP call resilience. It is not a circuit-breaker, bulkhead, or general resilience-pattern adoption — start with retry/backoff only, and file follow-ups if the design pass surfaces a need for more.
• Not a replacement for Kubernetes-level restart/readiness handling — the design pass should explicitly decide the boundary between "K8s handles this" and "the application should retry this."

Definition of Done

• Design decision documented (which calls retry, what policy, why) in a new or existing doc.
• Retry logic implemented at the decided call sites, with deterministic unit tests (injected time, per docs/TIME_INJECTION.md).
• Any operator-tunable retry parameters follow the SystemSettings pattern, only if the design concludes they should be tunable.
• RBAC call-out per AGENTS.md TDD Discipline step 0: this is primarily an internal reliability mechanism, not a new user- or admin-visible surface — state explicitly in the PR whether any part of it (e.g. a retry-count setting) is admin-facing.
• Standard merge gate from AGENTS.md's Definition of Done applies in full.