Skip to content

Add rate limiting to chat and auth endpoints #25

Closed

Description

@prometheusgr

Context

Flagged during a review of standard enterprise functionality against the current stack (Phase 4d complete). No rate limiting or throttling exists anywhere in the backend today.

The gap

POST /api/agents/chat and the auth endpoints have no per-user or per-IP request throttling. Each chat request drives an LLM call (llama-server) and, when a tool fires, an embedding + pgvector query (doc-search) — the most expensive path in the system. A single misbehaving client (buggy frontend retry loop, a compromised token, a scripted loop) can saturate llama-server for every other user with no backend-side control to stop it. This is a resource-exhaustion / availability gap, not a data-integrity one — distinct from the guardrails work (#16), which validates content, not request volume.

Scope

  1. Decide the mechanism and where it lives. Options: in-process (e.g. slowapi/token-bucket per process) vs. a shared store (Redis) for correctness across multiple backend replicas. Given the retention CronJob precedent (docs/SECURITY.md's "Enforcement" section) of preferring K8s-native/shared-state correctness over per-replica state, evaluate whether an in-process limiter is acceptable for v1 (single replica assumption) or whether shared state is needed from the start — decide deliberately and write down why, same as the CronJob rationale.
  2. Scope limits per identity, not just per IP. The caller's user_id (from the validated JWT, never request input) is the natural key for /api/agents/chat; unauthenticated endpoints (login-adjacent) need IP-based limits instead.
  3. Distinguish chat/tool-calling endpoints from cheap ones. A blanket global limit is the wrong shape — /api/agents/chat needs a tighter budget than, say, GET /api/agents/threads.
  4. Admin-configurable limits, following the established pattern (env default + nullable DB override, resolved per-call, audit-logged on change) already used for retention (Backups: strategy, encryption, and retention reconciliation #12-adjacent), guardrails (Guardrails: make thresholds and heuristics configurable #16), and LLM/embedding provider config — don't invent a new config shape.
  5. Response shape on trip: 429 with a Retry-After header, consistent with how the input guardrail already returns 400 with a reason code rather than a silent failure.
  6. Document in docs/SECURITY.md alongside the existing Guardrails & Compliance section — this is adjacent risk-mitigation, and a reader auditing abuse controls should find both in one place.

Standards

AGENTS.md. TDD as usual; the limiter's core logic (token bucket / sliding window math) is exactly the kind of deterministic, time-dependent logic that should accept now: datetime per docs/TIME_INJECTION.md, not call datetime.now()/time.time() internally.

Done when

Chat and auth endpoints reject excess requests with 429, limits are admin-configurable and audit-logged on change, the mechanism's replica-correctness tradeoff is documented, and docs/SECURITY.md reflects the new control.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions