Comparing JSON Ignoring Volatile Fields in Python Tests

A test that compares two JSON documents will fail for the wrong reason as soon as the documents contain something that changes on its own. Generated ids, creation times, request ids, ETags, random tokens: every one of them differs between runs, and none of them is the thing the test is about. The comparison has to ignore them without ignoring everything else.

I tried the usual ways to do this on invoices that carry a UUID and a timestamp, and measured what they cost and what they miss.

Strip the keys, then compare

The simplest approach is to remove the volatile keys from both sides and compare what is left:

VOL = ("id", "created_at")

def norm(d):
    d = copy.deepcopy(d)
    for k in VOL:
        d.pop(k, None)
    return d

assert norm(a) == norm(b)

On 2,000 pairs of invoices it took 0.034 s and all pairs were equal. It does one thing and has no dependencies, and the failure message from pytest shows a clear dict diff. The copy matters: stripping in place would change the object under test.

For a document where the volatile key can appear at any depth, strip recursively:

def strip_keys(o, keys):
    if isinstance(o, dict):
        return {k: strip_keys(v, keys) for k, v in o.items() if k not in keys}
    if isinstance(o, list):
        return [strip_keys(x, keys) for x in o]
    return o

strip_keys(doc, {"updated_at"})
# {"a": {"items": [{"id": "u1", "n": 1}, {"id": "u2", "n": 2}]}}

The input had updated_at at the root, in a and in both items, and all four were gone while the item ids stayed. The jq equivalent, handy in a shell test, is del(..|.updated_at?), and it gave {"a":{"n":2},"b":[{"n":4}]} for a similar document, as did walk(if type=="object" then del(.updated_at) else . end).

The risk with stripping by name is that it removes every key of that name, including a meaningful one at another level. If id is volatile at the top and meaningful in items, strip by path.

DeepDiff with exclusions

The deepdiff library compares with exclusion options and reports what differed:

from deepdiff import DeepDiff

assert not DeepDiff(a, b, exclude_paths=["root['id']", "root['created_at']"])
assert not DeepDiff(a, b, exclude_regex_paths=[r"root\['(id|created_at)'\]"])

An empty DeepDiff is falsy, so assert not DeepDiff(...) reads naturally. The path form targets one location, and the regex form matches many. Both were slower than stripping: 0.69 s and 0.68 s for 2,000 pairs, about twenty times the cost, because DeepDiff does a lot more per comparison. That is irrelevant for a hundred tests and noticeable for a hundred thousand.

What it buys is the report. When a test fails, the output says what changed and where:

{
  "values_changed": {"root['total']": {"new_value": 32.0, "old_value": 31.0}},
  "iterable_item_added": {"root['lines'][2]": {"sku": "C3", "qty": 1, "amount": 1.0}}
}

That was a real difference: I added a third line to one invoice, and with the volatile fields excluded, DeepDiff still found the added item and the changed total. Ignoring two fields did not make the comparison blind.

The options that matter for JSON are ignore_order=True ([1,2,3] equals [3,2,1], and a list of objects equals the same objects in another order, both of which I checked) and math_epsilon for floats. By default 0.1 + 0.2 and 0.3 were different, and with math_epsilon=1e-9 they were equal. Float tolerance is the second most common cause of flaky JSON tests, after timestamps (comparing JSON arrays where order does not matter explains the ordering choices).

Placeholders that keep the relations

Stripping loses information. If a document has the same id in two places, say a parent reference, the test should still show that they are equal. Replace values with numbered placeholders, in order of first appearance:

def placeholders(o, mapping=None, key="id"):
    mapping = {} if mapping is None else mapping
    if isinstance(o, dict):
        return {k: (mapping.setdefault(v, f"<id{len(mapping)+1}>")
                    if k == key and isinstance(v, str) else placeholders(v, mapping, key))
                for k, v in o.items()}
    if isinstance(o, list):
        return [placeholders(x, mapping, key) for x in o]
    return o

placeholders({"id": "u9", "parent": {"id": "u9"}, "kids": [{"id": "u3"}, {"id": "u3"}]})
# {"id": "<id1>", "parent": {"id": "<id1>"}, "kids": [{"id": "<id2>"}, {"id": "<id2>"}]}

Both appearances of u9 became <id1> and both of u3 became <id2>. Two documents built from different random ids now compare equal if they have the same structure of ids, and unequal if one has the parent pointing at a different id than expected. That is a stronger assertion than ignoring the field.

Where the exclusions should live

Where the list of volatile fields is kept decides whether it stays honest. A helper in the test module, next to the tests that use it, is read every time someone changes a test, and a constant at the top of a shared file is read never. I keep one small function per document type (invoice_for_compare) that returns the normalized value, and the tests call it on both sides. When the schema gains a new generated field, the failing tests point to exactly one place to change, and the review shows whether the new field was added to the ignored set on purpose.

It also helps to make failures readable. A comparison that fails on a 200 line document should print the difference, not both documents. With pytest, assert norm(a) == norm(b) prints a dict diff, and for large ones DeepDiff(...).pretty() returns a sentence for each change, such as Item root['lines'][2] added to iterable, which a person can scan in seconds.

Choosing what counts as volatile

Keep the list of volatile fields short, explicit and in one place, because every entry is a hole in the test. Some rules that have worked:

• Name them in the test, not in a global config that nobody reads.

• Prefer excluding by path over by name.

• Assert the shape of what you excluded: the id is a string that looks like a UUID, created_at parses and lies within the last minute. That way the field is checked, only not for equality.

• Make the source deterministic where you can. A fixed clock and a seeded id generator remove the need to ignore anything, and are better in the end.

For snapshots the same ideas apply as matchers and scrubs (snapshot testing JSON output). To see how two documents differ while you write the test, the JSON diff tool is faster than reading a failure message. The deepdiff options are documented at zepworks.com/deepdiff, the del and walk filters in the jq manual, and the wider topic of tests that produce JSON is in testing code that produces JSON.

When ignoring a field hides the bug

The field is a real contract. If the id format matters to consumers, assert on its format.

The field changed when it should not have. An updated_at that moves on a read request is a bug that an ignore list would hide. Test that the timestamp does not change, then ignore it elsewhere.

Everything is excluded. If the list of exclusions grows to half the document, the test compares very little. Rewrite it to assert on the parts that matter.

The exclusion is wider than the volatile value. Stripping by name removes legitimate fields of the same name at other levels. Use paths for nested documents.