Contract Testing JSON APIs With an OpenAPI Schema Tool

An API test usually asserts that the requests someone thought of produce the responses someone expected. A contract test asks a different question: does the service keep the promises written in its own specification, for requests nobody thought of? When the specification is machine-readable (an OpenAPI document), a tool can generate the requests, send them and check every response against the schema.

I pointed one such tool at a small FastAPI service with a few bugs planted in it, to see what a generated contract test finds and what it cannot.

The service and its contract

The models are Pydantic classes, so the contract comes free:

class Item(BaseModel):
    sku: str = Field(min_length=1, max_length=20)
    qty: int = Field(ge=1, le=1000)
    price: float = Field(ge=0)

class OrderIn(BaseModel):
    customer_id: int = Field(ge=1)
    items: list[Item] = Field(min_length=1, max_length=50)
    note: Optional[str] = Field(None, max_length=200)

FastAPI turns these into the JSON Schema inside /openapi.json: required fields, integer ranges, string lengths, array bounds. It documents two responses per operation, the success and a 422 validation error. I added two planted bugs. /search computes 100 // limit with no check that limit is not zero, and GET /orders/{id} raises when a stored note contains a NUL character.

Running the generated tests

Schemathesis reads the schema and builds requests from it, using Hypothesis underneath:

schemathesis run http://localhost:8123/openapi.json --max-examples 300 --checks all

The run produced 1,903 test cases in 4.5 seconds and found four unique failures in three operations. The one I expected:

GET /search
- ST101 Server error
    Received: 500   Documented: 200, 422
    curl -X GET 'http://localhost:8123/search?q=&limit=0'

limit=0 is a valid integer by the schema, so the tool tried it, and the division by zero returned a 500. Nobody had written that test. It is the usual kind of find: an edge of the input space that the types allow and the code does not handle. The fix is a constraint in the contract (limit: int = Query(10, ge=1, le=100)), which then also tells every client what is allowed, and turns the failure into a documented 422.

Each failure came with a curl command that reproduces it and a short id for st replay, which is the feature that makes the output usable. The seed is printed, so a run can be repeated.

Failures that are really contract bugs

The other three findings were not crashes:

GET /orders/{oid}   Received: 404   Documented: 200, 422
POST /orders        Received: 400   Documented: 201, 422     (a body that is not JSON)

The first is an order that does not exist, which returned 404 as it should. The OpenAPI document does not list a 404, so the tool reports an undocumented status code. The same for a body of random bytes, which the framework rejected with a 400 before the model validation ran. These are correct behaviors that the contract does not mention. The right fix is to document them (responses={404: {...}}), because a client generated from this contract would not handle a status that the contract does not list.

This category is the value of the exercise even when nothing crashes. The contract and the behavior drifted apart, and a generated client library built from the first would be wrong about the second. Ask for the 404 in the schema, and every consumer, human or generated, knows about it (versioning JSON payloads covers how to change a contract that clients already depend on).

What it missed

The second planted bug was not found. A note containing a NUL byte is accepted by POST /orders, stored, and makes GET /orders/{id} raise. To reach it a test must create an order with a particular note, then fetch the id the server returned. Schemathesis generates each operation's requests on its own. The OpenAPI document would have to declare a link from the response of POST /orders to the oid parameter of GET /orders/{oid} for the stateful phase to chain them, and FastAPI does not generate links. Without them the tool fetches random ids and gets 404s, which it already reported.

That is the limit of a schema-driven test. It checks each operation against the schema for every input the schema allows. It does not know what your operations mean together. Add a few explicit tests for the workflows that matter (create, read back, update, delete), and consider declaring OpenAPI links so the stateful phase can follow them.

A smaller version of the same idea

You do not need a tool to start. The core of a contract test is a schema and a loop:

schema = load("order.schema.json")
for sample in glob("contracts/order/*.json"):
    assert Draft202012Validator(schema).is_valid(json.load(open(sample)))

Keep sample responses of each endpoint in a folder, validate them against the schema in CI, and run the same schema check against the live response in an integration test. The consumer side can do the reverse, with a schema of only the fields it reads, and assert that the provider's sample responses satisfy it. That is the idea of consumer-driven contracts, which tools such as Pact formalize; I did not run Pact here. A provider that changes a field breaks the consumer's schema in the provider's build, not in production (event payloads in JSON and schema evolution has the compatibility matrix for which changes break which kind of consumer).

Running it in CI

A few practical points from the run. Start the service on a fixed port with a throwaway database, as the test creates data. Pin a seed in CI so a failure can be reproduced, and use a random seed in a nightly job to explore. Set a per-operation example count that keeps the run to a minute. Fail the build on server errors (5xx) always, and on undocumented statuses once the contract is clean. Point the tool at your staging environment only with the data safety in mind: it sends nonsense to every endpoint, including the destructive ones.

For the request side of the same problem, see validating request JSON before it reaches the database, and for the wider test picture testing code that produces JSON. Schemathesis is documented at schemathesis.readthedocs.io, and the OpenAPI specification at spec.openapis.org. To write the JSON Schema for a sample response, use the JSON schema generator.

When a generated contract test is not enough

The behavior is stateful. Anything that depends on earlier calls (a cart, a workflow, a rate limit) needs explicit scenario tests or declared links.

The schema is loose. A contract that says "object" or "string" without bounds generates weak tests. The tool is only as sharp as the schema, so tighten the constraints first.

The danger is in the data, not the shape. A request that is valid by the schema and wrong for the user (the wrong customer id) passes every generated check. Authorization needs its own tests.

The service has side effects you cannot undo. Sending generated requests to an endpoint that charges cards or sends emails needs a sandbox or a mock. Exclude the operation from the run rather than hoping.