Handling JSON Files Too Large to Parse With Ordinary Tools

At some size a JSON file stops being a file you open and becomes a problem you manage. The editor hangs, the script that has worked for a year dies with MemoryError, a colleague's laptop swaps for ten minutes and gives up, or a cloud function is killed with no message but a line in a log. Nothing is wrong with the file. JSON is a text format, and every tool that "parses" it builds a structure in memory that is several times larger than the text. The structure is what runs out.

This guide is the method I now use for files from a few hundred megabytes up to several gigabytes. It starts with why parsing fails and where the limits are, then goes through the steps in the order I take them: look at the file, put it in a streamable shape, query it, process it in code, find a record, split it, shrink it, and store it. Each step has measurements from the same data so the choices can be compared. The shorter articles linked from each section go into the detail.

Why parsing fails

A parser does not keep your text. It converts it to objects: a dictionary for each JSON object, a list for each array, a string object for each string and an integer object for most numbers. Each carries a header, pointers and padding. One event is 149 bytes of text and about 1,533 bytes as Python objects. For a whole array the result is a steady ratio:

file (array)    records      Python json.load     peak / file
  41 MB           250,000        328 MB              8.0x
  82 MB           500,000        645 MB              7.9x
 164 MB         1,000,000      1,279 MB              7.8x

Node does better on this shape, about four times (628 MB resident for the 164 MB file), because V8 shares the layout of objects with the same keys. A faster parser does not change the ratio: orjson finished the 164 MB file in 1.8 s instead of 2.9 s and peaked at 1,814 MB, higher, since it holds the input bytes while it builds the result (why loading a large JSON file eats all your memory has the full table of seven parsers).

So a gigabyte of JSON wants somewhere between 4 and 8 GB of free memory before the first line of your code runs, and the machine, container or function you are running in often has less. Three different walls show up:

python json.load, 984 MB array, 4 GB limit     MemoryError after 10 s
orjson.loads, same                             "Not enough memory to allocate buffer for parsing"
jq '.[0]' on the array                         jq: error: cannot allocate memory
node fs.readFileSync(path, "utf8")             ERR_STRING_TOO_LONG (limit 536,870,888 characters)

The Node error is not about RAM at all. V8 refuses a string longer than about 512 MiB, so JSON.parse cannot even be offered a 984 MB document. And jq reads the whole input before it filters, so even .[0] needs all of it.

What does not hit the wall is anything that handles one record at a time. Reading the same 6,000,000 records line by line used 12 to 18 MB, whatever the file size. That is the whole idea of this guide: get to a shape where a record is the unit, and let the tools that work per record, or per chunk, do the job.

Step 1: look before you load

Never start by opening the file with the tool that fails. Look at its shape with tools that read a few bytes:

ls -l big.json                  # how big, really
head -c 300 big.json            # does it start with [ or { ?  one line or many?
tail -c 100 big.json            # does it end properly, with ] or }
file big.json                   # text, with very long lines?

Three questions matter, and the answers decide the rest.

Is it a single array, an object, or one record per line? A leading [ means an array: parse it with a streaming reader or convert it. A leading { followed by a newline with another { on the next line is almost certainly JSON Lines. A file that starts {"data":[ is an array inside an envelope, and the path to the records is data.item for a streaming parser. The first 300 bytes tell you in a millisecond.

Does it end where it should? A download that stopped at 90% ends in the middle of a value. The last 100 bytes show it, and no parser needed (repairing truncated JSON that was cut off covers recovering what is there). A streaming reader will deliver every complete record before the cut and then report the break, so the damage is usually the last few records, not the file.

Is it one line? A 984 MB file with one line defeats grep, head, sed, split and awk, because all of them think in lines. I measured grep -b -o on the one-line array: 9.6 s and 1.9 GB of memory, since grep loads a whole line. wc -l returning 0 or 1 for a big file is the sign. Everything that follows is easier on a file with one record per line, which is why the next step exists.

Step 2: get it into lines

JSON Lines (also called NDJSON): one complete JSON value per line, no enclosing brackets, no commas between. A million records cost the same bytes as the array (the two forms I measured differed by 2 bytes), and nearly every large-data tool reads it incrementally (JSON Lines for data pipelines and log files). If you produce the data, write it this way and skip the rest of this step. If someone else produced an array, convert once.

import orjson
from stream import iter_array    # the 25-line raw_decode reader from the streaming article

with open("big.json") as f, open("big.ndjson", "wb") as out:
    for rec in iter_array(f):
        out.write(orjson.dumps(rec))
        out.write(b"\n")

On the 984 MB array this took 25.2 s with 25.7 MB of memory, and the output was byte-for-byte identical to the original JSON Lines file (cmp reported no difference), which also confirms that the array was read whole: 6,000,000 records, none lost, none repeated. Reading alone, without writing, took 11.6 s. The reader is the standard library's raw_decode called on a growing buffer, and it needs no packages (streaming a giant JSON array without loading it). With ijson the same read took 2.6 s for 164 MB with its C backend and 38 s on its pure Python fallback, so check ijson.backend before trusting it. jq --stream also streams, with flat 2 MB memory, but took 44 s for the 164 MB file where the plain filter takes 5.6 s, so it is for peeking and for files that fit nothing else:

jq -c --stream 'select(length==2) | .[0] | join(".")' big.json | head -3

For a conversion you will run repeatedly, the details are in converting a JSON array to JSON Lines and back. If the file has an envelope, say {"meta":{...},"items":[...]}, give the streaming reader the prefix (items.item in ijson) and read the metadata separately.

One caution on correctness: the conversion re-serializes, so compact output may differ from the source in whitespace and in how numbers are written. orjson and the standard module both preserve key order and write 5.0 as 5.0; very long integers and NaN differ between libraries (how much faster orjson is than the standard library lists them). If the data holds integers beyond 64 bits, convert with the standard library, not orjson.

When the top level is an object, not an array

Some exports are one giant object keyed by id, {"1": {...}, "2": {...}}. The same rule applies and the same tool handles it: ijson.kvitems(f, "") yields each key and value pair as soon as its value is complete. I built a 172 MB object of 1,000,000 records and read it both ways:

json.load                    3.62 s    1,388 MB     1,000,000 keys
ijson.kvitems(f, "")         2.77 s       18 MB     1,000,000 keys, 250,000 purchases

The streamed read was faster and used one seventieth of the memory. The loaded dictionary was bigger than for the equivalent array, since a million key strings are objects too. For an envelope with several large members, call ijson.items or kvitems once per prefix, which reads the file once per call; if that is too slow, ijson.parse yields every event in one pass and you pick out what you need.

Step 3: query it without writing code

Many "large file" questions are questions: how many, how many of these, which ones, what is the distribution. A database engine that reads JSON answers them faster than a script. DuckDB reads JSON Lines in parallel, infers the columns and lets you ask in SQL:

duckdb -c "select count(*), sum((event='buy')::int) from read_ndjson('big.ndjson')"

6000000, 1500000      0.78 s    296 MB

On one thread, 2.4 s with 79 MB. It also reads the one-line array directly (3.1 s, 276 MB), which no line tool can, and it reads gzip and zstd files by extension (querying JSON files with DuckDB, reading newline-delimited JSON in DuckDB). A few more checks you can run before deciding anything else:

select event, count(*) from read_ndjson('big.ndjson') group by 1 order by 2 desc;
select min(ts), max(ts), count(distinct "user".id) from read_ndjson('big.ndjson');
describe select * from read_ndjson('big.ndjson');

The last one shows the schema DuckDB inferred, which is a fast way to see what the data looks like without reading it. Two cautions: type inference samples the beginning of the file by default, so a field that is a number for the first 20,000 rows and a string later can make the read fail or mistype; and DuckDB re-serializes what it writes, so do not use it to rewrite the file if you need the original text intact (it changed 2026-10-02T01:01:07Z to 2026-10-02 01:01:07+00 in a test).

For a plain count or a text match on JSON Lines, grep is faster than anything that parses: grep -c '"event":"buy"' found the 1,500,000 matches in 1.1 s with 2 MB. It matches text, not fields, so a nested key with the same name would count too, and a writer that put a space after the colon would be missed.

Step 4: process it in code

When the work is more than a query (call an API for each record, clean values, write to several places), read the lines in your language. I ran the same job, parse every line and count the 1,500,000 buys, with the common choices:

approach                            time      memory
DuckDB, all cores                   0.78 s    296 MB
orjson, 8 processes                 1.79 s     20 MB
msgspec, typed decoder              3.55 s     18 MB
orjson, text prefilter              4.05 s     18 MB
msgspec, generic                    5.41 s     18 MB
orjson                              6.29 s     18 MB
ujson                               7.50 s     13 MB
Node, readline + JSON.parse         8.43 s     89 MB
python-rapidjson                   12.49 s     15 MB
standard json                      15.87 s     13 MB
jq 1.8.2                           20.06 s      2 MB
pandas read_json, 500k chunks      27.48 s  1,791 MB

Three lessons, each from benchmarking JSON parsers on a 1 GB file, which has the whole table and the code. First, the standard library is a respectable baseline at 62 MB/s and sixteen seconds a gigabyte; a faster parser buys a factor of 2.5, not 20. Second, doing less work wins more than parsing faster: a typed msgspec decoder that declares only the two fields it reads, and an orjson loop that parses only lines containing "event":"buy", were both quicker than parsing everything with the fastest generic parser. Third, a file of independent lines parallelizes by byte range: give each worker an offset, have it skip to the next newline, and it processes complete lines only. Eight workers gave 3.5 times.

The loop itself should not accumulate. Count, aggregate into a small dictionary, write each result out, insert in batches. A list that collects every record rebuilds the problem you just removed. And decide what a bad line does before the job runs: raise, skip and count, or write to a reject file. A million-line file with a stray unescaped quote is not a hypothetical.

Step 5: find one record

The most common request is the smallest: find the record with this id. The answer depends on the shape.

grep -m1 '"id":5999990,' big.ndjson | jq .

The last of 6,000,000 records came back in 1.3 s with 1.4 MB of memory; one near the start took under 10 ms because -m1 stops at the first match. Including the quote and the trailing comma matters so that id 10 does not also match id 100. jq did the same scan correctly but took 18 s, and DuckDB 1.15 s with exact field matching. In a program, test the bytes first and parse only candidates: 3.1 s, 5 times faster than parsing everything.

For many lookups, build an index of byte offsets once (5.4 s, 106 MB for 6,000,000 lines) and seek straight to a record: the lookup took 0.21 s including reading the index with sed, and under a millisecond with the offsets in memory. At that point a database is the better answer, as the next section shows. All of this, including the one-line trap, is in finding one record in a multi gigabyte JSON file.

Step 6: split it

Splitting turns one awkward file into several workable ones, for parallel workers, for upload limits, for retries. On JSON Lines it is the oldest tool there is:

split -l 1000000 -d --additional-suffix=.ndjson big.ndjson part_      # equal record counts, 3.0 s
split -n l/8      -d --additional-suffix=.ndjson big.ndjson part_      # equal bytes, whole lines, 1.7 s
split -C 128m     -d --additional-suffix=.ndjson big.ndjson part_      # size cap, whole lines, 2.4 s

The one flag to avoid is -b, which cuts at an exact byte and splits a record in two; I saw one part end in "page":"/p/19"," and the next begin with ms":619}}. Adding --filter='zstd -q -3 -o $FILE.zst' compresses each part as it is written, so the whole gigabyte was cut and compressed in 1.7 s. A JSON array cannot be split by line tools; it needs the streaming reader (34.6 s for six valid array files, with every boundary checked) or DuckDB, which is quick (11 s) but needs 2.6 GB and rewrites timestamps. Convert first, then split. The details are in splitting a large JSON file into smaller ones.

Step 7: make it smaller

Whatever else happens, the file will be stored, copied and fetched more often than it is written, so shrink it. The levers, in the order of effort:

Compress it. The 183 MB test file became 18.7 MB with gzip -6 and 18.4 MB with zstd -3; zstd compressed it in 0.3 s against 1.8 s and decompressed it in 0.15 s against 0.65 s. On the 984 MB file zstd decompressed in 0.8 s and gzip in 3.6 s. Zstd at level 9 reached 18.9 times in 0.5 s with threads, and level 19 gave 26.6 times for 29 s and 685 MB of memory. A pipe from zstdcat into grep counted the matches in 0.9 s, faster than zcat at 4.0 s. Use zstd for files you control, gzip when unknown readers must open the file (gzip versus zstd for JSON files).

Write compact JSON. Pretty printing made 250,000 events 67% bigger than compact (68.1 MB against 40.7 MB), but after gzip the difference was 24%, and shortening the keys saved 18% raw and 5% compressed. Whitespace is worth removing for parse time and for uncompressed paths, and rarely worth more effort (reducing JSON size by dropping whitespace and repeated keys).

Change the shape. A columnar JSON layout, one array per field, compressed to 0.34 MB with zstd against 4.58 MB for the same records as compact JSON. That is the idea behind the next lever.

Change the format. MessagePack saved 29% raw, but after gzip it was within 1% of JSON, and orjson decoded JSON as fast as MessagePack decoded its own. Parquet is where the large gain is: 784,820 bytes with zstd for the 250,000 events, 52 times smaller than the JSON Lines, and a count over it took 0.04 s against 0.16 s on the text (binary alternatives to JSON and when they pay off, converting JSON to Parquet with DuckDB). Parquet is the format to archive data you will query; JSON stays the format you exchange.

Step 8: put it in a database

If the questions will keep coming, load the data into something with indexes. PostgreSQL's COPY reads JSON Lines straight into a jsonb column, because each line is one value:

CREATE TABLE ev (doc jsonb);
COPY ev FROM '/tmp/events5.ndjson';

1,000,000 events (183 MB of text)   COPY 2.95 s     table 261 MB
count where doc->>'event'='buy'     250,000 rows    43 ms
lookup by id, no index              Parallel Seq Scan, 53.7 ms
CREATE INDEX ((doc->>'id')::int)    271 ms          index 21 MB
lookup by id, with the index        Bitmap Index Scan, 0.045 ms

A million records loaded in three seconds, and the lookup that scanned the table in 54 ms took 45 microseconds with an expression index. COPY in its default text format treats the backslash as an escape, and a line such as {"msg":"say \"hi\""} fails with invalid input syntax for type json. I tested the workaround: COPY ... WITH (FORMAT csv, QUOTE E'\x01', DELIMITER E'\x02') uses two control characters that do not occur in JSON text, and loaded the escaped strings and a Windows path intact. The benchmark data had no backslashes, and yours may. Once the records are in a table, the rest of the toolkit applies: working with JSONB in PostgreSQL, generated columns for the fields you filter on, and a conversion to relational columns when the shape is stable. If the documents belong in a document database, see moving between JSON and MongoDB documents for the import flags that keep the types.

For one-off analysis, skip the database: DuckDB on the file, or on a Parquet copy of it, is faster to set up and as fast to query.

What goes wrong, and what to do

**MemoryError or the process is killed.** The file was parsed whole. Stream it, or use JSON Lines with DuckDB. In a container, set a memory limit so that you get an error with a stack trace and not an out-of-memory kill.

**ERR_STRING_TOO_LONG in Node.** The file is over about 512 MB as a string. Read it as a stream and split lines, or process in Python or DuckDB.

**jq: error: cannot allocate memory.** jq loads the whole input. Use --stream, or convert to lines and run jq on those.

**Unexpected end of JSON input or a premature EOF.** The file is cut off. Check the tail, and stream the good records out.

The job is correct and slow. Count the cores you are using. A line file parallelizes by byte range; a one-line array does not.

The counts differ between tools. A grep count includes nested matches and a streamed count may stop at a corrupt line without saying so. Compare a record count from two methods before you trust either.

A UTF-8 byte order mark at the start. Some exporters write one, and some parsers refuse it. It is three bytes, visible with head -c 3 | xxd.

Estimating before you run

Do not discover the limit by crashing into it. Take a slice and measure. On a line file, head -n 100000 big.ndjson > sample.ndjson gives a sample in a second; for an array, stream the first thousand records out with the reader and wrap them in brackets. Load the sample with the approach you plan to use and read the peak memory (/usr/bin/time -v on Linux, -l on macOS). Multiply by the file size over the sample size, add a quarter for headroom, and compare with what the target machine has. For my events the 41 MB sample peaked at 328 MB, and the 984 MB file would have needed about 8 GB, which the 4 GB limit confirmed by failing.

The same sample is the place to try the parsers and the queries, since a 100 MB slice shows the ranking of approaches in seconds and the full file shows it in minutes. Keep the sample honest: take it from the middle or the end as well as the start, because formats drift (a field that appears in later records, a type that changes), and the first rows of a file are rarely the most irregular ones.

A worked example, end to end

A partner sends a 984 MB array, big.json. The questions: how many events, how many purchases, and what does event 5,999,990 look like. Then keep the data for later.

step                                            time       result
head -c 300, tail -c 100                         0.0 s      one-line array, ends with ]
convert to JSON Lines (raw_decode + orjson)     25.2 s      big.ndjson, identical to the lines form
count and filter with DuckDB                     0.78 s     6,000,000 events, 1,500,000 purchases
grep -m1 '"id":5999990,' | jq .                  1.3 s      the record
zstd -3 -T0 big.ndjson                           0.9 s      110,305,868 bytes, 8.9x
zstd -d to /dev/null (later reads)               0.8 s

The whole path costs about half a minute of compute, uses under 300 MB at its worst, and leaves a file that every later question can reach in a second or two. Doing it the first way anyone tries, json.load on the array, would have used about 8 GB and failed on most machines. The saving comes from the shape, not from any clever tool.

If the partner can be asked, the cheapest improvement is upstream: ask for JSON Lines, compressed with zstd or gzip. Nothing else in this guide is as effective.

Which step solves which problem

The steps are not a sequence you must complete. Most jobs need two or three:

the problem                                 start here
I need a count, a filter, a distribution    Step 3 (DuckDB on the lines)
I need to run code on every record          Step 4 (lines, orjson or msgspec, workers)
I need one record                           Step 5 (grep, then jq)
I need to send it in pieces                 Step 6 (split --filter)
it is too big to store or ship              Step 7 (zstd, then Parquet)
people will ask questions for months        Step 8 (database with an index)
it is an array and nothing else works       Step 2 first, always

Step 2 is the one the others depend on. Before it, only DuckDB and a streaming parser can read the file at all; after it, grep, head, split, awk, parallel workers and every database loader work.

A checklist

1. Check the size, the first 300 bytes and the last 100.

2. If it is an array, convert it to JSON Lines once, streaming.

3. Query with DuckDB before writing any code.

4. For code, read lines; use orjson or msgspec, prefilter, and parallelize by byte range.

5. Use grep -m1 for one record, an index for many, a database for always.

6. Split with split -l, -n l/N or -C, never -b.

7. Compress with zstd, and store query data as Parquet.

8. Validate the result: record counts match, last ids match, sample records compare.

9. Keep the original until step 8 passes.

When the file is not actually too large

It fits and you need the whole thing. A 200 MB file takes 1.6 GB in Python. If the machine has 16 GB and the job runs once, json.load is fine and shorter than anything above. Reach for the techniques here when the ratio puts you near the limit, or when the job runs on a small box, in a container with a cap, or in a function with a time limit.

The problem is a bad file. An error on a small file is usually syntax: a trailing comma, a missing quote, a BOM. Size is not the cause.

It is slow only because of what you do with it. A per-record network call or a quadratic loop makes a 20 MB file take hours. Profile before changing the reading.

You can ask for a better format. If you control the export, JSON Lines and compression solve most of this before it starts.

The limits quoted come from the Node.js buffer documentation and the formats from RFC 8259 for JSON; DuckDB's readers are described in its JSON documentation and PostgreSQL's COPY in the PostgreSQL manual. To read a record you pulled out, use the JSON formatter.