Mock Data Generators for JSON Tests Demos and Load

A test with no data tests nothing, and a test with hand-typed data tests the cases its author thought of. Generated data adds the volume and variety that a person will not write, and it brings its own problems: nondeterminism, collisions, and values that are valid but unrealistic. The right tool depends on what the test is for.

I tried four generators on the same kind of record, a user with an id, email, plan and optional fields, and noted what each is good at and where each fails.

Faker for realistic values

Faker produces names, emails, addresses and dates that look like data:

from faker import Faker
Faker.seed(42)
f = Faker()
{"name": f.name(), "email": f.email(), "city": f.city()}
# {"name": "Allison Hill", "email": "[email protected]", "city": "New Roberttown"}

Seeding is the point. Calling Faker.seed(42) again gave the same three values, and Faker.seed(7) gave Chris Curtis. A fixture generated with a recorded seed can be regenerated exactly, and a test that fails with seed 1234 can be rerun with seed 1234. Print the seed in the test output, or keep it as a constant, because an unseeded generator produces a failure you cannot reproduce.

Locales are one argument: Faker(["ja_JP", "de_DE"]) produced 渡辺 直樹, Univ.Prof. Halina Zirme B.A. and a Japanese address, which is the quickest way to find a column or a JSON consumer that mishandles non-ASCII text. Use real-looking data in at least one test, since [email protected] never exercises the same code paths.

Two limits showed up. Speed is modest: 10,000 rows with a name, an email and a date took 1.53 s, so a million rows takes minutes. And values collide: those 10,000 rows had 9,794 distinct emails and 9,377 distinct names. If a column has a unique index, or a test asserts a count of distinct users, random data will fail it sometimes. Faker's unique proxy exists for this, and it has a ceiling too:

fu = Faker()
for _ in range(3000): fu.unique.random_int(0, 1000)
# UniquenessException

Asking for 3,000 unique values from a pool of 1,001 raised an exception. For identifiers, do not use a random generator. Use a counter (user-0001, user-0002) which is unique by construction and readable in a failure message.

Hypothesis for the cases you did not think of

Faker gives plausible data. Hypothesis gives hostile data, and shrinks it when something fails. You describe the shape of a document with strategies:

from hypothesis import given, strategies as st

user = st.fixed_dictionaries(
    {"id": st.integers(1, 1_000_000), "email": st.emails(),
     "plan": st.sampled_from(["free", "pro", "team"])},
    optional={"seats": st.integers(1, 500),
              "tags": st.lists(st.text(max_size=8), max_size=3)})

Optional keys are real optional keys, absent from some examples. A sample it found: {"id": 1, "email": "[email protected]", "plan": "free", "seats": 1, "tags": ["", "", ""]}. Note what the tool favors: small numbers, empty strings, boundaries. These are where the bugs live.

I planted a bug in a function that normalizes a plan name, returning "TEAM " with a trailing space for one case, and wrote the property that the result equals plan.upper():

@given(user)
def prop(x): assert normalize_plan(x["plan"]) == x["plan"].upper(), x

Hypothesis found a failing input and reported the minimal one: {'id': 1, 'email': '[email protected]', 'plan': 'team'}. It began from a random large document and shrank it, removing optional keys and simplifying values, until nothing more could be removed. The shrunken example tells you what matters: only the plan.

A property that found real bugs

The more useful use is in tests of code you did not write. A round trip property says that writing a value and reading it back returns the same value:

json_values = st.recursive(
    st.none() | st.booleans() | st.integers() | st.floats(allow_nan=False) | st.text(),
    lambda c: st.lists(c, max_size=3) | st.dictionaries(st.text(), c, max_size=3),
    max_leaves=12)

@given(json_values)
def rt(x): assert json.loads(json.dumps(x)) == x

It passed 300 examples for the standard library's json, including with ensure_ascii=False. The same property for orjson failed, with two distinct failures. I reproduced each by hand:

orjson.dumps(float('inf'))      # b'null'          (json.dumps gives Infinity)
orjson.dumps(2**64)             # TypeError: Integer exceeds 64-bit range
orjson.dumps('\ud800')          # TypeError

inf is written as null, so the value silently changes, and integers beyond 64 bits and lone surrogates raise. Neither is a bug in orjson, which documents these choices. They are things your code must know before swapping it in for json (faster JSON in Python with orjson has the other differences). A property test finds those in a minute, where example-based tests find them in production.

Note the caution about scope: Hypothesis shows that no failing input was found in the examples tried. It does not prove correctness. Run it with a larger max_examples in a nightly job, and keep the database of failures it stores so that old counterexamples are replayed.

SQL and jq for volume

When you need a lot of data and not variety, the database and the shell are quicker than Python:

select jsonb_build_object('id', i, 'email', 'user'||i||'@example.com',
  'plan', (array['free','pro','team'])[1+i%3],
  'tags', case when i%2=0 then '["a"]'::jsonb else '[]'::jsonb end)
from generate_series(1, 1000000) i;

Deterministic, with no dependency, and the output goes straight into a table. jq does the same on the command line:

jq -nc '[range(100000) | {id: ., v: (. * 7 % 1000)}]' | wc -c
# 2077892   (0.18 s)

100,000 objects took 0.18 s. Neither gives realistic strings, and both give exactly the same data every time, which is what you want for load tests and for reproducing a query plan (indexing hot JSON properties for dashboards uses data like this). Mix: generate keys and numbers in SQL, and sprinkle realistic strings from a Faker-produced file.

Choosing

Keep generated data out of version control when it is large, keep the generator and its seed in, and validate generated documents against the same schema as production ones (validating request JSON before it reaches the database). The JSON schema generator can write that schema from a sample. The docs are at faker.readthedocs.io and hypothesis.readthedocs.io. For the larger picture of tests around JSON, see testing code that produces JSON.

When generated data is the wrong kind of test data

You are reproducing a bug from production. Use the real (anonymized) payload that caused it. A generator will not produce the particular combination, and the exact bytes matter.

The test asserts exact output. If it compares against a snapshot, random input makes it flaky. Seed the generator or use fixed fixtures (snapshot testing JSON output).

The data has to be legally safe. Generated data is safe by construction. Production copies are not, even after masking, unless you have checked the masking.

The schema is what is under test. For a contract, a handful of explicit valid and invalid documents is clearer than a thousand generated ones.