Guard against embedding dimension mismatch and trigger re-embedding on embedding model change #77

Context

docs/CONFIG_AUDIT_2026-09-04.md's RAG audit (Part 1, alongside #68) flagged an "adjacent landmine" found while confirming that EMBEDDING_DIMENSION = 768 (backend/app/services/embedding_service.py:16, duplicated in mcp-servers/doc-search/app/search.py:56) is correctly hardcoded — it's tied to the pgvector column's fixed width (Embedding.embedding: vector(768)), so it can't be a per-call SystemSettings override the way embedding_model itself already is.

The landmine: embedding_provider/embedding_url/embedding_model are already fully admin-configurable at runtime via the Settings UI (Phase 2b, resolve_embedding_config). Nothing today stops an admin from switching to a different embedding model whose output dimension differs from 768. The consequences of doing so are currently undefined behavior, not a handled error — likely an insert failure at write time (if pgvector rejects a mismatched-width vector) or, worse, silently wrong cosine-distance math if some code path doesn't validate width at all. Either way, existing indexed documents remain embedded with the old model's vectors, silently mixed with (or blocking) new ones — this is exactly the "silent mixing of incompatible vectors" problem Phase 3's EmbeddingResult(vector, provider, model) provenance tagging was built to make detectable, but detectability isn't the same as prevention or remediation.

Two related but distinct problems to solve

1. Prevent/guard the mismatch: validate that a newly configured embedding model's output dimension matches the schema's fixed 768 before accepting the settings change (or before the first write using it), rather than discovering it via a failed insert or corrupted similarity search.
2. Trigger re-embedding on model change: when an admin changes embedding_model (even to a same-dimension model — a different model can still produce vectors that aren't comparable to the old model's, even at the same width, since embedding spaces aren't interchangeable across models), existing documents' embeddings are now stale/incompatible and should be re-generated against the new model — not left silently mixed. This is the more significant part of the issue: it needs a re-embedding job, not just a validation check.

Scope

1. Dimension validation: when embedding_model/embedding_provider/embedding_url changes via PUT /api/settings, probe the new configuration (a single test embedding call, similar in spirit to the existing provider_probe_service.py idiom referenced in issue Surface Keycloak admin/login events in the EAIStack Audit Log view (read-only) #63's design) and reject the change with a clear error if the returned vector's dimension doesn't match the schema's fixed width. Fail loudly at configuration time, not silently at the next write.
2. Re-embedding trigger: on a successful embedding-model change (dimension validated), mark existing Embedding rows as stale (their embed_metadata provenance tagging from Phase 3 already records which provider/model produced each vector, so "stale" is computable as "provenance != current config" without a new column) and enqueue/trigger re-embedding of the affected knowledge-base documents.
3. Decide the re-embedding mechanism — this needs its own explicit decision, mirroring how Move rate-limit state to a shared store when the backend scales past 1 replica #38 documents the rate-limit store choice and Backups: strategy, encryption, and retention reconciliation #12 documents backups: a synchronous blocking re-embed on the settings-change request (bad for anything beyond a tiny KB), a background job similar in shape to the retention CronJob (infra/k3s/retention-cronjob.yaml), or an on-demand admin-triggered action ("Re-embed all documents" button) with progress visibility. Given re-embedding an entire knowledge base could take significant time depending on corpus size and the embedding provider's throughput, a fire-and-forget background approach with a status/progress indicator is likely the right shape, but confirm during implementation rather than assuming.
4. UI element (required): the Settings screen must show, when embeddings are stale (some or all documents' embed_metadata provenance doesn't match current config), a clear indicator and either an automatic re-embed trigger or an explicit "Re-embed now" action — silently leaving the knowledge base in a mixed/stale state with no visible signal would recreate exactly the blind spot Phase 3's provenance tagging was built to prevent.
5. Interaction with search during re-embedding: decide and document whether search continues to work (against a mix of old/new vectors — likely degraded relevance) during re-embedding, or is temporarily restricted — don't leave this unspecified.

Definition of Done

• Dimension-mismatch validation added to the embedding-config change path, with a unit test covering a rejected mismatched-dimension change.
• Re-embedding mechanism decided and documented (sync/background job/on-demand — pick one, write the rationale in docs/BACKEND_SERVICES.md or a new doc).
• Re-embedding implemented, using the existing embed_metadata provenance fields to identify stale rows rather than adding a redundant new column.
• Settings UI shows staleness state and a re-embed trigger/status, admin-gated same as the rest of Settings.
• Documented behavior for search-during-re-embedding.
• Unit tests covering: successful model switch + re-embed, rejected mismatched-dimension switch, and (if applicable) partial re-embed failure handling.
• RBAC call-out per AGENTS.md TDD Discipline step 0: admin feature, same as the rest of Settings/embedding config.
• Standard merge gate from AGENTS.md's Definition of Done applies in full (tests, lint, e2e for the new UI signal, docs — likely docs/SECURITY.md or a new docs/RAG_CONFIGURATION.md alongside Make RAG retrieval/chunking tuning knobs admin-configurable (similarity threshold, top_k, chunk size/overlap) #68).