Skip to content

Review OpenAPI drift test and response validation #61

Open

Description

@ctengel

The test suite gained two pieces of OpenAPI tooling (added under #34, commits 10a1a58 and 5491dce) that are worth a deliberate review later. Both work and pass today, but were scoped pragmatically.

  • tests/test_openapi_drift.py — asserts openapi.yaml and the FastAPI apps agree on which paths/methods exist.
  • tests/openapi_validation.py — ValidatingTestClient checks every test response's status code and JSON body against openapi.yaml.

Things to revisit:

  1. Drift test scope. It deliberately compares path + method coverage only, not status codes or response schemas — FastAPI's generated spec is too thin for that (see the module docstring). Decide whether that is the right permanent boundary.
  2. Validation approach. ValidatingTestClient is a hand-rolled wrapper using jsonschema. Consider whether a dedicated tool (schemathesis, openapi-core) is a better long-term fit, especially for request-side validation, which is currently not covered.
  3. Undeclared test dependencies. This work added respx and relies on jsonschema, referencing, and pyyaml. CLAUDE.md only lists fastapi/uvicorn/httpx/pytest, and there is no dependency manifest — these should be documented (or a manifest introduced).
  4. problem+json mismatch. The spec documents error responses as application/problem+json / ProblemDetail, but FastAPI actually returns application/json / {"detail": "..."}. The validator tolerates this only because ProblemDetail has no required fields. Decide whether to emit real RFC 9457 problem details or relax the spec.
  5. Duplicate operationId warnings. Pre-existing: api_route declaring GET+HEAD on one function produces duplicate operationIds, visible in test output. Cosmetic.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentation

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions