skills/api-contract-testing/SKILL.md (4887 bytes)
1 --- 2 name: api-contract-testing 3 description: Property-based testing of HTTP APIs against an OpenAPI or GraphQL specification using Schemathesis. Use when the project has an openapi.json, openapi.yaml, swagger.json, or an auto-generated spec (FastAPI /openapi.json, Django REST, Go/Java annotation-based specs); when adding or modifying an HTTP endpoint; when the spec and implementation may have drifted; or when a client reports a response shape the docs do not describe. 4 --- 5 6 # API contract testing 7 8 Schemathesis reads an OpenAPI spec, generates requests from the declared 9 schemas, and checks that responses conform. It finds the two failure modes 10 example tests never catch: inputs nobody thought to try, and drift between the 11 spec and the implementation. 12 13 ## Preflight 14 15 ```bash 16 command -v schemathesis || uvx schemathesis --help 17 schemathesis --version 18 ``` 19 20 The CLI has changed across major versions (option names and defaults differ 21 between v3 and v4). **Run `schemathesis run --help` and use what that reports** 22 rather than relying on remembered flags. Report the version in the output. 23 24 Install: `uv tool install schemathesis`, or run ad hoc with `uvx schemathesis`. 25 26 ## Locating the spec 27 28 ```bash 29 fd -e json -e yaml . | rg -i 'openapi|swagger' 30 curl -s localhost:8000/openapi.json | head -c 200 # FastAPI, Litestar 31 curl -s localhost:8080/v3/api-docs | head -c 200 # springdoc 32 ``` 33 34 If the spec is generated at runtime, point Schemathesis at the live URL so it 35 always tests the current contract. A stale checked-in spec tests nothing useful. 36 37 ## Baseline run 38 39 Start the service, then: 40 41 ```bash 42 schemathesis run http://localhost:8000/openapi.json --url http://localhost:8000 43 ``` 44 45 Default checks cover: undocumented status codes, response bodies that violate 46 the declared schema, wrong content types, malformed headers, and server errors 47 (any 500 is a failure). Consult `--help` for the current check list and for how 48 to select or exclude individual checks. 49 50 Useful controls (verify names against `--help`): 51 52 - Limit scope while iterating: filter by path or method rather than running the 53 whole surface on every change. 54 - `--max-examples` — raise for a thorough run, lower for a fast inner loop. 55 - Authentication: `-H "Authorization: Bearer $TOKEN"`, or `--auth user:pass`. 56 - `--report` / cassette output — records failing interactions for replay. 57 58 ## Interpreting failures 59 60 Schemathesis reports a minimised failing case plus a `curl` command that 61 reproduces it. Triage into three buckets — the classification matters more than 62 the fix: 63 64 1. **The implementation is wrong.** Fix the code. Add the minimised case as a 65 unit test first, and show it failing. 66 2. **The spec is wrong.** Fix the spec. This is common and valuable: the spec is 67 what clients and codegen consume, so a wrong spec is a live bug even when the 68 server behaves correctly. Do not loosen the spec merely to silence a failure 69 — that discards the signal. 70 3. **The generated input is genuinely invalid** and the spec failed to say so. 71 Tighten the schema (`minLength`, `pattern`, `format`, `enum`, `minimum`). 72 This is the best outcome: the spec becomes more precise, and downstream 73 codegen and validation improve for free. 74 75 A 500 on any generated input is always bucket 1. Malformed input must produce a 76 4xx, never an unhandled exception. 77 78 ## Stateful testing 79 80 Schemathesis can chain requests using OpenAPI `links` to test sequences 81 (create → read → update → delete), catching bugs that single-request tests miss. 82 This requires `links` to be declared in the spec. Check current syntax with 83 `--help`; if links are absent, adding them improves both the docs and the tests. 84 85 ## In the test suite 86 87 Prefer running Schemathesis in-process against the ASGI/WSGI app rather than 88 over the network — faster, no port juggling, works in CI without a service 89 container. The Python API for this has changed between major versions; consult 90 the installed version's documentation rather than assuming the decorator name. 91 92 Whichever form is used, drive it against the app object, seed any required auth 93 via fixtures, and commit every discovered failing case as an ordinary example 94 test so it stays covered after the fuzzing budget moves on. 95 96 ## CI 97 98 Run a bounded pass on every PR (low `--max-examples`, fail on any error) and a 99 longer scheduled run nightly. Treat spec changes as API changes: if a PR alters 100 the spec, say so in the summary, and check whether the change is 101 backwards-compatible for existing clients. 102 103 ## Not an OpenAPI project? 104 105 - GraphQL: Schemathesis supports GraphQL schemas directly. 106 - gRPC/protobuf: use `buf breaking` for compatibility checks plus property 107 tests over the generated types. 108 - No spec at all: consider whether one is warranted. If the API is internal and 109 small, plain property tests over the handler functions are cheaper and give 110 most of the benefit.