Finding One Bad Line in a Huge JSON Lines File Fast

A load stops with "malformed JSON" and a line number, you open the file at that line, and the line looks fine. Or you run jq on a 36 MB file, it exits with an error that points to a line in the middle of nowhere, and you have 200,000 candidates. The way out is to stop asking one parser to read the whole file and instead parse every line on its own, then print the numbers of the ones that fail. This article shows how to do that in four tools, what the reported line number really means, and how to separate good lines from bad ones so the rest of the file can proceed.

Why the reported line is wrong

Run the obvious command first:

jq -c . events_bad.ndjson > /dev/null

jq: parse error: Expected separator between values at line 123458, column 1

The real fault is on line 123456, two lines earlier. The reason is how a parser reads a stream. jq treats the file as a sequence of JSON values separated by whitespace. A truncated object, {"id": 123456, ...,"user": {"id":, is not an error at its end, because the parser has not seen the end yet: the next line's opening brace is read as a continuation, and the parser only fails when it hits something that cannot continue the structure. So the error is reported where the parser gave up, not where the data went wrong. DuckDB reads one line at a time and reported at byte 242 in line 123457, one line off, since it counts from the start of the record that failed to close.

The practical rule is that a line number from a whole-file parse is a hint. The faulty line is at that number or somewhere before it, and the second claim is the one to remember for truncated records. A stray comma behaves differently: the second fault at line 150001 ({...},) is only detectable when the parser reads that line as a separate value.

Parse each line on its own

The reliable method gives every line its own parse. In jq, read the file as raw text with -R and call fromjson on each line, catching failures:

jq -R -c 'select((try fromjson catch null) == null) | [input_line_number, .[0:60]]' events_bad.ndjson

[123456,"{\"id\": 123456, \"ts\": \"2026-10-05T00:36:00Z\", \"user\": {\"id\": "]
[150001,"{\"id\": 150001, \"ts\": \"2026-10-06T01:01:00Z\", \"user\": {\"id\": "]

Both seeded faults, with their real line numbers and the start of each line, in 0.80 s. Two details. input_line_number counts lines read so far, which is the current line when you use -R. And the condition == null would also match a line whose value is the JSON literal null; if null is legal in your file, test for failure explicitly with select(try (fromjson | false) catch true).

The same idea in Python gives the reason too:

import json, sys
for n, line in enumerate(open(sys.argv[1], "rb"), 1):
    try:
        json.loads(line)
    except json.JSONDecodeError as e:
        print(n, e.msg, "col", e.colno, line[:40])

123456 Expecting value col 1 b'{"id": 123456, "ts": "2026-10-05T00:36:0'
150001 Extra data col 176 b'{"id": 150001, "ts": "2026-10-06T01:01:0'

The messages classify the faults: Expecting value at column 1 is the truncated object, Extra data at column 176 is the trailing comma after a complete value. Reading bytes ("rb") avoids a decoding error on a bad byte stopping the loop for the wrong reason. This took about the same time as the jq version on 200,000 lines, and the error text is the Python parser's, which is what a Python loader will report later. For a decoded explanation of each message, see fixing JSON parse errors when reading files.

Use the shape of the damage

If you know how files break, a one-line filter finds suspects in a few hundredths of a second. Each line of this file ends with }, so a line that does not is a candidate:

awk '!/}$/ {print NR": "substr($0,1,50)}' events_bad.ndjson

It listed lines 123456 and 150001 immediately, and grep -vn '}$' took 0.05 s. This is a heuristic and not a validator. It finds a truncated line and a trailing comma, but misses a line that ends in } and has an unquoted key in the middle. Use it to look quickly, and the parse-each-line method to be sure. Trailing carriage returns (\r) also fail the pattern, which will flag every line of a CRLF file, so strip them first or match }\r?$.

Bisect with head

When the only tool you have is one that fails on the whole file, bisect: head -n N file | tool. With a bad line at 123456, head -n 100000 and head -n 123455 both exited 0, and head -n 123456 exited 5. Binary search needs about 18 steps for 200,000 lines, and each is one command. It is slower than the per-line parse, but works with a validator you cannot loop over, such as a database loader.

DuckDB hides bad lines when told to ignore them

DuckDB can read the file and skip errors:

SELECT count(*) FROM read_ndjson('events_bad.ndjson', ignore_errors = true, columns = {id: 'BIGINT'});

It returned 200,000. All the lines are still there: with an explicit column list, a line that cannot be parsed becomes a row of NULLs, so the count equals the line count, and the bad lines are only visible as id IS NULL. Without ignore_errors, the same query stopped with Malformed JSON in file ... at byte 242 in line 123457. If you use ignore_errors, count the NULL ids, or you will report a clean load of a damaged file (see reading newline-delimited JSON in DuckDB).

Split the file into good and bad

Once you know there are bad lines, keep working with the good ones and put the bad ones aside to look at:

jq -R -c 'try (fromjson | tojson) catch empty' events_bad.ndjson > good.ndjson
jq -R -n -c 'inputs | select(try (fromjson | false) catch true)' events_bad.ndjson > rejects.txt

good.ndjson had 199,998 lines and rejects.txt had 2. Note that fromjson | tojson normalizes spacing: lines come out compact, and an integer like 12345678901234567890 is kept as written by jq 1.8 but can be rounded by older jq versions, so check numbers if they matter (converting a JSON array to JSON Lines and back). For a byte-for-byte copy of the good lines, use the line numbers from the first command and sed -e '123456d;150001d'.

Other causes of a failing line that is not truncated

• A byte order mark on the first line. In a test, jq stripped it and parsed the file, and Python's json.loads on bytes detected it, but a strict reader that decodes as UTF-8 and parses text will fail at line 1 (JSON parse fails with a byte order mark).

• A blank line. jq skipped it, and Python's json.loads raised Expecting value: line 2 column 1 (char 1). Strip blank lines or have the loop continue on them.

• Two records on one line. The Extra data error above. It usually comes from a writer that missed a newline (appending to a JSON Lines file safely).

• A raw newline inside a string, from a producer that did not escape it. The record splits across two lines, and both halves fail.

When the line is not the problem

The whole file is wrong. If most lines fail, the file may be a JSON array, a different encoding or a pretty-printed document. Look at the first few bytes with head -c 200, not at the failing lines.

Only a type check fails. A line that parses but has "ms": "slow" where numbers are expected is valid JSON. The loader that complains about it is reporting a schema problem, so use the type-checking approach in validating JSON with a schema in Python.

It is one bad line out of billions. On a pipeline that sees bad lines every day, stop hunting and make the loader write rejects to a side file, with the line number, so they are reviewed in batches.

For the format itself, see jsonlines.org and the jq manual on fromjson and error handling. The JSON formatter will show exactly where a copied line breaks.