Snapshot Testing JSON Output Without Flaky Failures

A snapshot test stores the output of a function the first time it runs, and from then on fails if the output changes. For JSON it is the cheapest regression test there is: you do not write the expected value, you review it once. That works until the output contains something that changes on every run, and then the test fails for no reason and gets deleted. Most flaky snapshot suites are unmasked volatile values.

I wrote a function that builds an invoice with a random id and a current timestamp, and snapshot-tested it with pytest and with Node's runner to see what fails and what fixes it.

The first run passes

def test_invoice_exact(snapshot):
    assert build_invoice(7, LINES) == snapshot

With --snapshot-update, syrupy wrote __snapshots__/test_snap.ambr with the output in Python's repr style:

# name: test_invoice_exact
  dict({
    'created_at': '2026-10-27T04:22:59+00:00',
    'id': '6a690803-f831-4547-a2e9-c184cde14670',
    'lines': list([
      dict({'amount': 19.0, 'qty': 2, 'sku': 'A1'}),
      dict({'amount': 12, 'qty': 1, 'sku': 'B2'}),
    ]),
    'order_id': 7,
    ...

(I have folded the nested dicts onto single lines here; the file puts every key on its own line.) Two things in that file are worth noticing before the second run. The values are Python values, so a float shows as 19.0 and an int as 12, and the function returned an int for the second line because round(1 * 12, 2) returns 12. That is a real quirk of the code, now recorded in a file, and a reviewer can see it. And the id and timestamp are in the snapshot as literals.

The second run fails

E   AssertionError: assert [+ received] == [- snapshot]
E     -   'created_at': '2026-10-27T04:22:59+00:00',
E     +   'created_at': '2026-10-27T04:23:02+00:00',
E     -   'id': '6a690803-f831-4547-a2e9-c184cde14670',
E     +   'id': '4e448c0e-8c98-4118-b937-bfe5849a78e6',
1 snapshot failed. 2 snapshots passed.

The diff names exactly the two values. The failing test was the one with no handling, and the other two passed in the same run. This is the classic failure, and the temptation is to run --snapshot-update and move on, which makes the test a coin toss that always passes after an update and so asserts nothing.

Fix one: a matcher

Syrupy lets you describe which values to compare by type instead of by content:

from syrupy.matchers import path_type

def test_invoice_matcher(snapshot):
    inv = build_invoice(7, LINES)
    assert inv == snapshot(matcher=path_type({"id": (str,), "created_at": (str,)}))

It stored the type, not the value:

'created_at': str,
'id': str,

and the test passed on the next run, and the one after. The matcher checks that id is a string and then ignores which string. The cost is that a change from a UUID to an integer shows up (as a failure on the type) and a change from one UUID format to another does not. If the format matters, scrub the value to a placeholder instead, or add a separate assertion that the id matches a UUID pattern.

Fix two: scrub before comparing

The other approach is to replace volatile values with fixed placeholders before the comparison, in your own helper:

def scrub(d):
    d = json.loads(json.dumps(d))
    d["id"] = "<uuid>"
    d["created_at"] = "<ts>"
    return d

def test_invoice_scrubbed(snapshot):
    assert scrub(build_invoice(7, LINES)) == snapshot

The snapshot then reads 'created_at': '<ts>' and 'id': '<uuid>', which is self-explanatory to anyone who opens the file. I prefer this for anything that is checked by reading, because the intent is visible. The round trip through json.dumps and json.loads ensures the helper does not mutate the original. When ids are repeated across a document and must be consistent (the same user appears twice), replace them with numbered placeholders so that the equality relation is kept; comparing JSON while ignoring volatile fields shows a helper that does it.

Even better is to control the sources of volatility: pass a fixed clock and an id generator into the function, so the output is deterministic and nothing needs to be scrubbed. That is a code change that pays back elsewhere, but not every function allows it.

A format that reviewers can read

By default syrupy stores Python reprs. A snapshot that is also JSON is better for a service that returns JSON, because reviewers see the same text the API returns, and other tools can read it:

from syrupy.extensions.json import JSONSnapshotExtension

@pytest.fixture
def snap_json(snapshot):
    return snapshot.use_extension(JSONSnapshotExtension)

def test_json_snapshot(snap_json):
    assert build_invoice(7, [("A1", 2, 9.5)]) == snap_json(
        matcher=path_type({"id": (str,), "created_at": (str,)}))

The file __snapshots__/test_json_ext/test_json_snapshot.json held:

{
  "created_at": "str",
  "id": "str",
  "lines": [{"amount": 19.0, "qty": 2, "sku": "A1"}],
  "order_id": 7,
  "tags": ["gift", "web"],
  "total": 19.0
}

with the matched fields written as the type names. Note the JSON snapshot sorts keys, so reordering fields in the code does not fail the test, which is usually what you want.

Node's built-in snapshots

Node 24 has snapshot assertions in node:test:

test('scrubbed snapshot', (t) => {
  const inv = build(); inv.id = '<uuid>'; inv.created_at = '<ts>';
  t.assert.snapshot(inv);
});

Running with node --test-update-snapshots --test created snap.test.mjs.snapshot, containing the value serialized as indented JSON text. The scrubbed test passed on every later run. The unscrubbed one failed with + actual - expected lines for the id and the created_at, the same pattern as in Python. Numbers print the JavaScript way (19, not 19.0), so a snapshot is also a statement about the language that produced it, and a cross-language contract needs a schema and not a snapshot (contract testing JSON APIs).

Habits that keep snapshots useful

• Read the snapshot diff in review. An update accepts the new output, so the diff of the snapshot file is the test. A 400 line snapshot change nobody read is no test at all.

• Keep snapshots small. Snapshot the part of the response that the test is about. A whole 2,000 line document is something nobody will review.

• Name them. A test called test_invoice_exact that was unfixable is a smell; the name should describe the situation.

• Do not auto-update in CI. Run with the update flag only on a developer's machine.

• Prefer a few explicit asserts for small outputs. A snapshot of three fields is often clearer as assert result["total"] == 31.0.

Related pieces: comparing JSON while ignoring volatile fields for the matching side, testing code that produces JSON for the wider picture, and writing tests that compare JSON output from the diff cluster. The syrupy documentation is at syrupy-project.github.io, and Node's snapshot API is in the test runner documentation. To view a snapshot as a tree, paste it into the JSON formatter.

When a snapshot is the wrong assertion

The output is tiny. One or two values read better as explicit assertions.

Almost everything in it is volatile. A document that is mostly timestamps and generated ids leaves nothing stable to snapshot. Assert on the few fields that carry meaning.

The shape matters more than the values. For a contract between services, a schema check says what must hold and tolerates everything else. A snapshot fails when anything changes.

Nobody will review the diff. A snapshot that is regenerated on every failure is worse than no test, because it looks like coverage.