Make RAG retrieval/chunking tuning knobs admin-configurable (similarity threshold, top_k, chunk size/overlap) #68

Context

An audit of the RAG/embedding pipeline (see docs/CONFIG_AUDIT_2026-09-04.md, Part 1) found that several operationally-relevant retrieval/chunking knobs are hardcoded Python constants instead of following the repo's established SystemSettings pattern (env-default + nullable-DB-override, resolved per-call via a resolve_*_config function, exposed on the Settings UI with a tooltip, audit-logged on change) already used for retention, guardrails, tracing, and rate-limiting.

Unlike llm_provider/embedding_provider/embedding_model (already fully configurable via system_settings_service.py), nothing about how retrieval works is currently tunable without a code change and redeploy.

Gaps found

1. Similarity/relevance threshold — missing entirely, not just hardcoded. EmbeddingRepository.search_similar (backend/app/repositories/embedding_repository.py:17-40, duplicated in mcp-servers/doc-search/app/repositories/embedding_repository.py:53-80) does ORDER BY distance LIMIT top_k with no WHERE distance < X cutoff. A query with zero relevant documents in the corpus still returns the k least-bad chunks — there is no way today to have the system say "nothing relevant was found."
2. Max results (top_k) — mcp-servers/doc-search/app/server.py:147, mcp-servers/doc-search/app/search.py:169, backend/app/mcp_client/doc_search_client.py:76 all default to 5, duplicated three times, with no server-side ceiling — the LLM can request any top_k it wants via the tool call.
3. Chunk size — backend/app/services/chunking_service.py:23-24, MIN_CHUNK_TOKENS=500 / MAX_CHUNK_TOKENS=1000, module constants.
4. Chunk overlap ratio — backend/app/services/chunking_service.py:30, CHUNK_OVERLAP_RATIO = 0.125, module constant.
5. Excerpt truncation cap — mcp-servers/doc-search/app/search.py:24, MAX_EXCERPT_CHARS = 2000, module constant. Governs how much of a retrieved chunk's text reaches the LLM per source.
6. Two duplicated/inline candidate-pool multipliers (mcp-servers/doc-search/app/repositories/embedding_repository.py:23 _CANDIDATE_MULTIPLIER = 4, and an un-named inline payload.top_k * 4 at backend/app/api/embeddings.py:72) are lower-priority internal implementation details, not admin-facing — at minimum, the inline one should become a named constant even if not made configurable.

Why bundle these together

These knobs are reviewed and tuned together in practice (a change to chunk size interacts with excerpt cap and top_k; a similarity threshold interacts with top_k), and they share one Settings UI section and one design conversation (schema fields, retroactivity caveats). Splitting them into separate issues would fragment that conversation.

Scope

1. Add nullable SystemSettings columns: rag_similarity_threshold (float, None = no cutoff — preserves today's behavior), rag_max_results (int, becomes doc-search's actual enforced ceiling, not just a suggested default), rag_chunk_size (int), rag_chunk_overlap_ratio (float), rag_max_excerpt_chars (int). Follow docs/DATABASE_MODELS.md's shape and the existing retention_hours/guardrail_* nullable-override precedent exactly.
2. New resolve_rag_config following docs/BACKEND_SERVICES.md and the existing resolve_guardrail_config/resolve_retention_config shape, resolved per-call (not at startup — unlike tracing_enabled, none of these require a process restart to take effect).
3. Wire rag_similarity_threshold into EmbeddingRepository.search_similar in both backend/ and mcp-servers/doc-search/ as a WHERE distance < threshold (or equivalent) predicate.
4. Wire rag_max_results into doc-search's search_knowledge_base (app/server.py) as the actual enforced LIMIT, not just a caller-suppliable default.
5. Wire rag_chunk_size/rag_chunk_overlap_ratio into chunking_service.py, replacing the module constants.
6. Wire rag_max_excerpt_chars into doc-search's search.py, replacing MAX_EXCERPT_CHARS.
7. Settings UI: new "Retrieval & Chunking" section on the Settings screen, each field with a tooltip, following the existing info-tooltip.tsx pattern from the Settings Help Text work.
8. Audit-log every change ("rag_config.config_update"), following the existing before/after-diff pattern ("retention.update", "guardrail.config_update").
9. Document the retroactivity caveat explicitly in the Settings UI copy and in docs/SECURITY.md (or a new docs/RAG_CONFIGURATION.md, matching docs/RETRIEVAL_IMPROVEMENT_PROMPTS.md's existing shape): changing chunk size/overlap only affects newly-(re)indexed documents, not documents already chunked and stored — unlike the LLM/embedding provider switch, which applies to the very next call. Consider whether existing documents should show a "needs re-chunking" indicator (can be scoped out to a follow-up if so).

Non-goals

• No reranking (already tracked separately — see docs/RETRIEVAL_IMPROVEMENT_PROMPTS.md's "Prompt 4," not yet a numbered issue).
• No change to the RRF fusion constant (RRF_K = 60) or hybrid-search candidate multiplier — low-value, rarely-tuned literature-standard defaults; out of scope here.
• No prompt-text changes (the search_knowledge_base tool description / chat agent system prompt are a separate, differently-shaped gap — free text, not a scalar field — and need their own design, not bundled here).

Definition of Done

• Five new nullable SystemSettings columns + Alembic migration.
• resolve_rag_config service function, unit-tested against [None, 0, boundary] per field per AGENTS.md's retention-field-semantics pattern where applicable.
• All five knobs wired into their actual enforcement points (not just resolved and ignored).
• Settings UI section with tooltips; component-tested.
• Audit logging on change.
• Retroactivity caveat documented in both UI copy and a docs file.
• RBAC call-out per AGENTS.md TDD Discipline step 0: this is an admin feature, gated by require_admin, same as Settings.
• Standard merge gate from AGENTS.md's Definition of Done applies in full (tests, lint, e2e for the new UI, docs).