Splitting a JSON Lines File Into Chunks Without Breakage

There are good reasons to cut a large JSON Lines file into pieces: a loader that times out on big files, an upload limit, a pipeline that can process eight chunks at once, a colleague who needs a sample. JSON Lines makes this easy, because every line is a complete record, and so the only rule is to cut at line ends. Most of the ways people split files do not follow that rule. These are the ones that do, what each one produces on a real file, and a short check that proves the pieces add up to the original.

Split by line count

The common case is an exact number of records per file:

split -l 50000 -d --additional-suffix=.ndjson events.ndjson chunks/part-

-l 50000 writes 50,000 lines per file, -d uses numeric suffixes (00, 01, ...) instead of aa, ab, and --additional-suffix adds the extension so the files are recognized by their tools. The result was part-00.ndjson to part-03.ndjson, 200,000 lines in total, in 0.07 s. This is the right choice when a downstream limit is expressed in records, such as an API that accepts at most 1,000 documents per request (use -l 1000 there). The files differ in size when the lines do, so the last one is smaller whenever the line count is not a multiple of the chunk size.

Split into a fixed number of chunks

When you want, say, one chunk per CPU, ask for a count of chunks, and tell split to respect lines with the l/ prefix:

split -n l/4 -d --additional-suffix=.ndjson events.ndjson chunks/q-

l/4 means four chunks, split on line boundaries. The output sizes were 9,107,723, 9,107,769, 9,107,654 and 9,107,542 bytes, within 0.002% of each other, while the line counts were 50,185, 50,122, 49,847 and 49,846. So l/N balances by bytes: it picks the byte offset where each chunk should end and moves to the next newline. That is what you want for parallel work, because the time to process a chunk follows its size. If you ask for -n 4 without the l/, the chunks are exactly equal in bytes and the cuts fall wherever they fall, which is the same failure as -b below.

Split by size

For an upload limit in megabytes, -C (line bytes) puts as many whole lines as fit into each file:

split -C 10M -d --additional-suffix=.ndjson events.ndjson chunks/c-

The 36 MB file gave chunks of 10,485,673, 10,485,748, 10,485,702 and 4,973,565 bytes. Each is at most 10 MiB, and each ends on a newline. I parsed every chunk with jq -c . and none failed. Note the unit: 10M is 10 × 1024 × 1024 bytes, and 10MB is 10,000,000. If the target system counts in decimal megabytes, write -C 10000000. One exception: a single line longer than the limit is split at the limit. I fed it a 300 byte line with -C 100 and it came out as three pieces of 100 bytes, so a record bigger than your chunk size is cut in the middle. Check the longest line first (awk '{ if (length($0) > m) m = length($0) } END { print m }').

The split that corrupts

split -b cuts at a byte count:

split -b 10M -d --additional-suffix=.ndjson events.ndjson chunks/b-

It writes the same four sizes, and each chunk ends in the middle of a record, so the next one starts in the middle of one. All four chunks failed jq -c .. For a text format with a line terminator there is no reason to use -b, and it is the form that turns up in old shell snippets. If you see a split command with -b on JSON Lines, replace it with -C. The same goes for head -c, dd bs= and any byte-offset slicing: they will produce truncated records at the edges (finding one bad line in a huge JSON Lines file is what you will end up doing afterwards).

Compress while splitting

--filter runs a command per output chunk, with the chunk written to its standard input and the name available in $FILE:

split -l 50000 -d --filter='gzip -6 > $FILE.gz' events.ndjson chunks/g-

The four outputs, g-00.gz to g-03.gz, were 896,459, 896,481, 896,316 and 896,227 bytes. No uncompressed intermediate was written. cat chunks/*.gz | zcat | wc -l printed 200,000: concatenated gzip members form a valid gzip stream, so the parts can also be fed straight to a loader that reads stdin (see streaming JSON Lines into PostgreSQL). The single quotes around the filter matter, since otherwise your shell expands $FILE to nothing. Choices of compression level and codec are covered in compressing JSON Lines for storage and reading them back.

Split by a key in the data

Sometimes the chunk is a partition: one file per day, per customer, per event type. split cannot do that, and awk can:

awk -F'"' '{print > ("chunks/day-" substr($6,9,2) ".ndjson")}' events.ndjson

Here the field separator is the double quote, so for this file's layout field 6 is the timestamp value and substr($6,9,2) takes the day of month. It produced 28 files (days 01 to 28) in 0.54 s. This is fast and fragile: it depends on the key order and spacing of the producer. A sturdier version extracts the key with jq and then hands the line to awk:

jq -r -c '"\(.ts[0:10])\t\(tojson)"' events.ndjson | awk -F'\t' '{print $2 > ("chunks/" $1 ".ndjson")}'

That is slower, because jq parses each line, and it is correct for any layout. Every partition is an open file while awk runs. GNU awk 5.2 coped with 28, and it copes with more by closing and reopening files behind the scenes, but other implementations (mawk, BusyBox) may stop at the open file limit. For a key with thousands of values, sort by the key first and close() each file when you leave it, or use DuckDB's COPY ... PARTITION_BY (converting JSON to Parquet with DuckDB).

Process the chunks in parallel

The point of splitting is often speed. I split the file into eight line-aligned chunks and ran the same jq filter, counting buy events, serially and with xargs -P 8:

ls chunks/p-* | xargs -P 8 -I{} sh -c 'jq -c "select(.event==\"buy\")" {}' | wc -l

Serial took 0.69 s and parallel 0.14 s, both printing 50,000, a speedup of about five times on a 10-core machine. The gain is below eight because the work is short and the processes compete for the same disk cache. Output from parallel jobs can interleave at line granularity when jobs write to the same pipe, which is fine for JSON Lines because lines are whole, but the order of the output is not the order of the input.

Prove the pieces add up

Two commands check any split. Count the lines in all the chunks (wc -l chunks/*, which printed 200,000 in total for each variant above) and compare checksums of the concatenation with the original:

cat chunks/r-* | md5sum
md5sum < events.ndjson

Both gave 1168606b077af501b94b844845b7e06a. Glob order matters here: the shell sorts part-00, part-01 and so on correctly. With -d, GNU split widens the suffix on its own when it runs out: splitting a 300 line file with -l 1 gave x-00 to x-89, then x-9000 onward, and the last file was x-9209. Those names still sort in creation order, but only because the prefix digits grow, so for more than 100 pieces set the width yourself with -a 4.

When splitting is the wrong fix

The tool can stream. DuckDB, jq and any line reader process the whole file in constant memory. Splitting adds files to manage without removing a limit. Try the tool on the full file first (reading newline-delimited JSON in DuckDB).

The file is one big JSON array. split -l on an array file produces fragments, not records. Convert it to JSON Lines first (converting a JSON array to JSON Lines and back), or follow splitting a large JSON file into smaller ones.

The bottleneck is a single database connection. Eight chunks loaded through one connection are no faster than one file. Parallelism pays when each chunk has its own worker.

The behavior of every option is in the GNU coreutils manual page for split, and the line format is defined at jsonlines.org. To check a suspicious chunk by eye, use the JSON formatter.