Compressing JSON Lines for Storage and Reading Them Back

JSON Lines is repetitive by design. Every line spells out the same keys, so a compressor can reduce a file to a small fraction of its size, and the files that pipelines produce are very often stored and shipped compressed. The questions are which codec to use and how to read the result back without first unpacking a copy to disk. I compressed one 36 MB file with the three codecs you are likely to have, timed the writing and the reading, and loaded the output with four different tools.

Size and time to write

Three things stand out. First, gzip's default level is already within 7% of its best (-9), which costs five times the time, so there is no reason to go beyond the default. Second, zstd at any level beat gzip's default on speed by a large factor (0.05 s against 0.32 s), and from level 1 on it also beat it on size. Third, zstd and xz can reach a 30x ratio, but at a time cost of half a minute for 36 MB. Levels above 9 are for archives you write once and read rarely.

The zstd rows contain a surprise. Level 3, the default, came out larger than level 1: 3,148,512 bytes against 2,745,098. Levels 2 and 4 were 2,851,913 and 3,141,559, and level 5 dropped to 2,635,470 and level 6 to 2,245,182. I repeated the runs and got the same numbers, so it is a property of the algorithm's strategy switch on this data and not noise. The lesson is general: compression levels are not guaranteed to be monotonic in size, and a 30 second test on your own file settles it (JSON Lines for data pipelines and log files discusses what makes these files compress so well). Use -T0 to let zstd use all cores on a bigger file; on this one it made no difference to the size (3,148,512 bytes) and took 0.03 s.

Time to read

Reading is cheap for all three. xz, the slowest to write by far, decompressed as fast as gzip. If a file is written once and read often, the write cost is paid once, so xz or zstd at a high level makes sense for archives. For a pipeline stage that writes a file every few minutes, speed on write counts and zstd at its default is the practical choice.

Reading without unpacking

You rarely need an uncompressed copy on disk. Each of these read the gzip file directly:

zcat events.ndjson.gz | jq -c 'select(.event=="buy")' | wc -l

That printed 50,000 in 0.68 s, which is the cost of the jq filter and not of the decompression. For DuckDB the file name is the whole story:

SELECT count(*) FROM read_ndjson('events.ndjson.gz');

It returned 200,000 in 0.22 s and 95 MB of memory. The extension tells DuckDB which codec to use, and it read events.ndjson.zst as well, with the same count. In pandas, compression='infer' is the default and works from the extension:

df = pd.read_json("events.ndjson.gz", lines=True)   # (200000, 7)

With the standard library, gzip.open("events.ndjson.gz", "rt") returns a text file object that iterates over lines, and counted 200,000. For zstd in Python you need a package such as zstandard or, from Python 3.14, the compression.zstd module; pandas can read .zst when zstandard is installed. For loading a compressed file into a database, see how zcat is piped into COPY in streaming JSON Lines into PostgreSQL, and for pandas details reading JSON Lines with pandas.

Searching a compressed file is slower than searching a plain one: zgrep -c for the buy events took 0.13 s against 0.03 s for grep -c on the raw file. That is a factor of four on a small file, and it is also the price of every query that cannot skip data. If you scan the same file often, keep a decompressed copy, or move it to a columnar format (converting JSON to Parquet with DuckDB).

Appending and chunking

Gzip members concatenate. Writing three batches of 1,000 lines with gzip >> m.gz gave a file that zcat read back as 3,000 lines, and gzip -l listed it as one stream. That makes append-only archives easy, and the cost is that each member has its own header and its own compression state, so small appends compress worse than one big write. Zstd has the same property. For storage, add data in batches of at least a few thousand lines and not one record at a time. When you split a file, compress each part: split -l 50000 --filter='gzip -6 > $FILE.gz' writes compressed chunks in one pass (splitting a JSON Lines file into chunks).

Minify first? Mostly no

I stripped the spaces after : and , with jq -c . (the file shrank from 36,430,688 to 32,390,688 bytes) and compressed that: gzip gave 3,446,753 bytes and zstd 3,134,529, against 3,582,540 and 3,148,512 for the original. The saving was 4% for gzip and 0.4% for zstd. A compressor already encodes the repeated ": " almost for free. Minifying is worth it when the file will be read uncompressed (and it makes each line shorter), but not as a way to save storage.

Small records and dictionaries

The ratios above come from a big file with a lot of repetition. Compressing each record on its own is a different problem, because there is little to repeat within 180 bytes. I took 500 single-record files and compressed each with zstd -3: 79,041 raw bytes became 68,748, only 13% smaller. Training a dictionary with zstd --train on 5,000 records and using it with -D ev.dict brought the same 500 to 21,906 bytes, 72% smaller. (zstd warned that the training set was small for the default dictionary size, so treat the number as an indication of the effect and not a figure to plan with.) A dictionary helps when you must compress or store records individually, as in a key value store, and every reader then needs the exact dictionary.

When compression is not the answer

The file is already compact. Binary-heavy or already compressed data (images in base64 excepted) gains little. Check with zstd -3 on a sample before building a pipeline around the ratio.

You need random access. A gzip file cannot be read from the middle. Splitting into chunks, a seekable format or Parquet gives access to parts of the data without scanning from the start.

Every consumer must be simple. If an old tool in the chain only knows plain text, the cost of an extra decompression step in every script may be larger than the storage saved. Gzip is the one codec every system can read.

The gzip format is specified in RFC 1952, the zstd command line and levels are documented at facebook.github.io/zstd, and DuckDB's handling of compressed JSON is on its JSON loading page. The JSON formatter handles the decompressed text when you need to read a record.