Testing Code That Produces JSON From Asserts to Contracts

Most code that produces JSON is tested by comparing its output to a literal. It works for the first test, and then the fifth test needs a timestamp, the tenth needs a generated id, and the twentieth is a 400 line expected document that everyone updates by pasting whatever the code now prints. At that point the tests assert nothing, and the only thing they do is fail at inconvenient times.

This guide is about doing better with a set of techniques that each solve a specific problem: assertions on parts, snapshots, comparing with exclusions, schemas and contracts, generated data, database fixtures and property tests. I ran each on small examples, and then did something I find more useful than any description: I changed the output in ten realistic ways and checked which kinds of tests noticed.

What there is to get wrong

A function that produces JSON can be wrong in a handful of distinct ways, and a test only helps if it notices the kind of wrong that is likely.

The shape can change: a key is renamed, dropped or added. The types can change: a number becomes a string, an integer becomes a float. The values can be wrong: a total that does not match its lines, a date in the wrong time zone. The format of a value can change: a timestamp in a different layout. The meaning of absence can change: a field that was null is now missing. And the text can change without the data changing: key order, whitespace, escaping.

Each of those may or may not matter to the consumer, which is the question every test of JSON output quietly answers. A consumer that parses the document ignores key order and whitespace. A consumer that compares raw bytes, signs the body, caches by hash or diffs files in a repository does not. Decide which consumer you have before you pick the assertion.

The toolbox, in the order you will need it

Explicit asserts on the parts that matter

The oldest technique is also the one to reach for first. Parse the output and check the fields the test is about:

inv = build_invoice(7, LINES)
assert inv["order_id"] == 7
assert inv["total"] == 31.0
assert [l["sku"] for l in inv["lines"]] == ["A1", "B2"]

It states exactly what is being tested, survives unrelated changes and produces a failure message that names the field. It does not notice anything you did not assert. Two asserts on order_id and total caught five of ten mutations in my matrix, below.

Snapshots for the whole shape

A snapshot stores the whole output once and compares later runs against it. It notices everything that changes, including what you did not think to assert, at the price of failing on changes that do not matter. The condition for using snapshots is that the output is deterministic, or that the volatile parts are masked. The details, with the failure you get from a random id and the two ways of fixing it, are in snapshot testing JSON output. The short version: scrub volatile values to placeholders or match them by type, store the snapshot as JSON so a reviewer can read it, and read every snapshot diff in review.

Comparing with exclusions

When you compare two documents, one of which you built by hand, you often need to ignore a few fields. Removing them before an equality check took 0.034 s for 2,000 pairs of invoices and DeepDiff with exclusions took 0.68 s but gave a precise report of what differed. Placeholders that number the ids keep relationships between values visible. The options, including float tolerance and ignoring array order, are in comparing JSON ignoring volatile fields.

Schemas and contracts

A schema describes the allowed shape and tolerates everything else, which makes it the right assertion for a contract between a producer and a consumer. It is the technique to use for an API that other teams depend on. You can validate every response in the test suite against the published schema, and generate requests from the same document to find inputs that break the service: 1,903 generated requests found four failures in a three-route service in 4.5 seconds, and also showed the limit of a schema-driven test, which is that it knows nothing about workflows that span calls (contract testing JSON APIs).

Generated data and property tests

When the input has to be realistic, a seeded generator produces data that looks like production and is repeatable. When the input has to be nasty, a property-based tool generates the boundary cases and shrinks a failure to the smallest example. A property that a JSON round trip returns the same value passed 300 examples for the standard library and failed for orjson on inf and on integers beyond 64 bits (mock data generators for JSON). That is a test you would not write by hand, because you would not think of the input.

Fixtures for tests that touch a database

If the code under test reads or writes rows, the test needs rows. A JSON fixture loaded with jsonb_populate_recordset loaded 25,000 rows in 0.214 s against 2.42 s for row-by-row inserts, let the table constraints reject bad fixtures, and rolled back at the end of each test to keep them isolated. It also showed a silent failure: a misspelled key is ignored (fixtures in JSON for database tests).

The experiment: ten changes, six checks

To compare these techniques on something other than argument, I took one function and wrote ten mutations of its output, each a change that a real refactor or a library upgrade could introduce. Then I wrote six checks of increasing strictness and ran every check against every mutation. A check "catches" a mutation if it fails on it. I also ran each check against the unchanged output to make sure none of them raised a false alarm, and none did.

The checks:

• exact text: json.dumps(output) == json.dumps(expected)

• == dict: Python equality on the parsed objects

• 2 asserts: order_id == 7 and total == 31.0

• schema: required keys and types, with date-time format checking

• schema + sum: the same, plus the rule that total equals the sum of the line amounts

• strict schema + sum: as above with additionalProperties: false

The results:

mutation         exact text  == dict  2 asserts  schema  schema+sum  strict+sum
rename key       caught      caught   caught     caught  caught      caught
string total     caught      caught   caught     caught  caught      caught
drop key         caught      caught   caught     caught  caught      caught
extra key        caught      caught   -          -       -           caught
reorder keys     caught      -        -          -       -           -
float noise      caught      caught   caught     -       -           -
null->missing    caught      caught   -          -       -           -
date format      caught      caught   -          caught  caught      caught
wrong total      caught      caught   caught     -       caught      caught
qty float        caught      -        -          -       -           -

Several things stand out.

Nothing catches everything except the strictest. Exact text caught all ten, because it compares the bytes, and for that reason it also breaks on harmless changes, and I would not use it for anything but a document consumed as text. It is in the matrix as the upper bound.

Equality on parsed objects misses two kinds of change. Reordering keys is invisible (correctly: dict equality ignores order). And a quantity changing from 2 to 2.0 was not caught, because in Python 2 == 2.0. Whether that matters depends on the consumer. A JavaScript client parses both as the number 2. A Go client decoding into an int fails on 2.0. A strongly typed consumer or a schema with integer is the place where it hurts, and JSON Schema counts 2.0 as an integer, so even the schema did not flag it. If the type of a number matters, assert on the text of the value or on isinstance(x, int).

A schema is the right tool for shape and the wrong one for values. It caught the renamed key, the string total, the dropped key and the bad date, which are the changes that break consumers. It passed the total that disagreed with its lines, a float total that was off by 1e-9 and a document that had lost its optional note. Adding a one-line rule (total == sum(amounts)) caught the wrong total. That rule is a business invariant, and invariants belong in tests next to the schema.

Closing the schema is a choice, not a default. Only the strict schema and the exact comparisons caught the extra key. The consumer that tolerates additions would not want that test, and a consumer that forwards documents to a strict third party would (event payloads in JSON and schema evolution covers which side you are on).

Null versus missing is easy to lose. Only the whole-document comparisons noticed that "note": null had become absent. If a consumer treats the two differently, that needs an explicit test.

Two explicit asserts are better than nothing and worse than they feel. They caught five of ten and passed the extra key, the missing null and the date format, which is what you would expect from asserting two fields.

The conclusion I draw is a combination: a schema for shape, a few explicit asserts for invariants, a reviewed snapshot for everything else, and a deliberate test for the cases where the text itself matters.

Floats, dates and other things that bite

A few specifics came up in the matrix and recur in practice.

Floats. A total computed as 0.1 + 0.2 is 0.30000000000000004. The mutation "float noise" added 1e-9 and only the exact comparisons noticed. Either compute money in integers or decimals, or compare with a tolerance (DeepDiff(math_epsilon=...), pytest.approx). Never snapshot a float that comes out of arithmetic without deciding how many digits matter.

Integers and floats. The qty float mutation is a reminder that the distinction between 2 and 2.0 exists on the wire and disappears in many comparison tools. Python's json.dumps(2.0) writes 2.0, PHP's json_encode(2.0) writes 2 unless told otherwise, and the same function in two languages produces different text (JSON in Go, Java, Rust, C#, PHP and Ruby compared shows the others).

Dates. A timestamp built from datetime.now() is volatile. One built from a fixed value is deterministic, but its text depends on the time zone and the formatting call. Freeze the clock in the test, or inject it, and assert the format with a regex or a schema format. The mutation "date format" was caught by any check that looked at the format and missed by those that looked only at the keys.

Key order. Serializers differ: Python keeps insertion order, jsonb in PostgreSQL sorts by key length then name, Go sorts map keys, and some libraries randomize. A test that compares text depends on that. A test that compares parsed values does not. Sort keys when you need stable text (sorting JSON keys for a clean version).

Unicode. json.dumps escapes non-ASCII by default and ensure_ascii=False writes the characters. Both parse to the same value. A test that compares text will notice a change in the setting, and a parsed comparison will not.

A refactor that looks harmless

The matrix is synthetic, so here is one that was not planned. While preparing the snapshot examples I changed build_invoice so that line amounts went through float(...), a tidy-up that makes every amount a float. Nothing about the invoice changed in meaning. The second line's amount went from 12 to 12.0 in the output.

Three of my tests reacted differently. The pytest snapshot failed, with a diff that showed - 'amount': 12 and + 'amount': 12.0. A plain equality of the two invoices, with the id and timestamp removed, passed, because Python considers 12 == 12.0. And DeepDiff with the same exclusions reported a difference by default, a change of type from int to float, and reported none when I turned on ignore_numeric_type_changes.

Is that change a bug? It depends on the consumer, which is the whole point. A browser reads both as 12. A consumer written in Go that decodes the field into an int would reject 12.0; the same Go decoder reads 12.0 into a float64 without complaint. PHP's json_encode would have written 12 for either unless told to preserve the fraction. A refactor that is invisible in one language changes the contract in another. The snapshot caught it and forced the question, and the equality test would have shipped it.

That is the argument for keeping at least one test that is sensitive to the text, for the documents that other systems read. It is also an argument for deciding the numeric type in the contract: an amount is a decimal with two digits, a quantity is an integer, and the schema says which. Then 12.0 for a quantity is a failure by definition, and not a matter of taste.

Choose the assertion by who reads the output

The question to ask before writing the assertion is who reads the document and how.

If a parser in another language reads it, assert the shape with a schema, and add explicit asserts for the values that drive behavior. Keep one snapshot for the awkward case. Types matter, so make the schema say integer or number deliberately.

If a human reads it (an export, a report, a log), the text matters. A snapshot of the file is the right test, reviewed like prose.

If the output is signed, hashed or cached by its bytes, then every character matters, including key order and whitespace. Canonicalize it (sorted keys, fixed separators) and test the canonical form exactly.

If it is an internal value passed between functions, test the Python or JavaScript object, and let the serializer be tested once.

If the output is stored and read back later, the test is the round trip. Write it with the real database, read it back, compare, and include the awkward values: a large integer, a decimal, a string with a quote and a newline, an emoji, a nested empty array. Most storage bugs are a value that comes back changed.

A review checklist for pull requests that touch JSON output

When a change touches code that produces JSON, a short list in the pull request template helps reviewers see what the tests do and do not cover. I would ask:

Does the change alter the set of keys, their types or their order? If so, is there a schema change in the same commit, and is the change compatible with existing consumers or versioned?

Does the change alter a number's text (integer to float, fewer digits, scientific notation)? Is there a test sensitive to it?

Do the new fields have a defined meaning when absent and when null? Is that tested?

Are any new values volatile (a timestamp, a random id, a counter)? Are they masked or injected in tests?

Did a snapshot change? Did a human read the diff, and does the description say why it changed?

Does the new behavior have a hostile-input test if it parses or transforms input?

None of these questions is hard, and together they catch the changes that tests alone cannot, because the reviewer is the only one who knows which consumers exist.

Structure the suite in layers

A layout that has worked for me, from cheap to expensive:

1. Unit tests of the builder, with fixed inputs and explicit asserts on the fields that matter. Fast, many, run on every save.

2. A schema check on every response in the suite. Add it to the test client so that every endpoint test validates its output for free. This is a few lines and catches the shape regressions across the whole API.

3. A small number of snapshots for the documents that other teams or systems consume, reviewed as code.

4. Property tests for the serialization boundary: round trips, parsers, anything that transforms JSON.

5. Contract tests generated from the published schema, run in CI against a test instance, with a seed pinned.

6. Database tests with fixtures, rolled back per test, for code that touches tables.

Keep the expensive ones few, and put the effort where the consumers are.

Put the schema check in the test client

The cheapest improvement for an API test suite is to validate every response without writing anything in the individual tests. In a test client wrapper, load the schema for the route once and run the validator after each call:

def get(client, path):
    resp = client.get(path)
    schema = SCHEMAS[(resp.request.method, route_of(path), resp.status_code)]
    jsonschema.validate(resp.json(), schema, cls=Draft202012Validator)
    return resp

Now each of the hundreds of existing tests asserts the shape as a side effect, and the day someone renames a field every test that touches it fails with the schema's message, which names the key. The cost is one lookup and one validation per call, which I measured at roughly twenty microseconds for a small schema in jsonschema, so it is invisible next to the HTTP round trip.

Two refinements help. Use the schema from the published OpenAPI document and not a copy, so that the test and the documentation cannot drift. And treat a missing schema for a route as a failure, so a new endpoint cannot be added without one. The same wrapper is the natural place to log the request id and the response body on failure, which saves the first ten minutes of every debugging session.

On the JavaScript side the same idea is a few lines with Ajv and a custom fetch helper, and in the other languages the pattern is the same: a validator behind the client. What matters is that the check is on by default, not a thing each test author has to remember.

Consumer-side tests

Everything above tests what you produce. The other half is what you consume. A consumer's test should feed it the documents it will really receive, including the awkward ones: a missing optional field, an explicit null, an unknown extra member, a number as a string, a duplicate key, a deeply nested value. Those tests are the same shape as the hostile bodies in JSON security mistakes in apps and pipelines, where fifteen bad bodies each returned a clean 4xx and none raised an exception. Store the bodies as files next to the code, name them for what they check, and add one whenever production surprises you.

Producers and consumers meet in the middle at the schema. A producer that validates its output against the schema in its own build, and a consumer that validates the producer's sample output in its own, are testing the same contract from both ends without needing to deploy together. When either changes the schema, one of the two builds fails, and the change is discussed before it ships.

Flaky tests and how to remove the cause

A test about JSON that fails one run in twenty almost always has one of four causes: a volatile value nobody masked (a timestamp, a random id), a dependency on key or array order that the code does not guarantee, a float that sums in a different order, or shared state between tests (rows left in a table, a global cache). The fixes are the ones already described: mask or inject the volatile value, compare parsed values or sort, use a tolerance, and roll back per test.

When you cannot find the cause, print what differs. A dict diff from pytest, the DeepDiff report, and the + actual - expected lines from a snapshot runner all show the first difference. Reading one failing diff tells you more than rerunning the test thirty times.

Keeping tests cheap to maintain

The cost of a JSON test suite is the cost of updating it. Snapshots that are updated without reading are expensive because they are worthless. Large literals that are edited by hand are expensive because they are error-prone. A few habits keep the cost down: build expected values from small helper functions instead of pasting a whole document; keep fixtures in files and shared across tests only where the data really is shared; name tests for the behavior; and delete a test that cannot say what it protects.

Run the layers in different places. The unit tests belong in the editor loop. Snapshots and schema checks belong in every pull request. Generated tests with large example counts and random seeds belong in a nightly job, with the failing seed saved in the report so that a failure can be replayed. And write down the decision about strictness once: whether the system tolerates unknown fields, whether null and missing are the same, and whether numbers are checked for their type. Every test in the suite is then a consequence of a stated rule and not an accident of how it was written.

The tools are documented at pytest, JSON Schema and Hypothesis. To inspect a failing document while debugging, the JSON formatter and the JSON diff tool are quicker than reading raw text. For the diff-focused side of testing, the earlier article on writing tests that compare JSON output covers a related set of choices.

When you are testing more than the output needs

The output is a thin wrapper. If a function only calls json.dumps on a dict you already tested, a test of the text adds nothing. Test the dict.

A framework guarantees it. A typed response model in a web framework serializes what the model says. Testing the model's types is testing the framework.

The consumer is you, today. A script that writes a JSON file you will read once does not need a contract suite. A single assert that it parses is enough.

The test costs more than the failure. A prototype or a throwaway export does not deserve six layers. Start with the schema check and the explicit asserts, and add the others when the output gets consumers.