Repairing Truncated JSON From a Cut Off Response or File

The response ends in the middle of a word. A log shows {"items": [{"id": 0, "name": "n0"}, {"id": 1, "nam and then nothing, and the parser reports that the input ended too early. Truncated JSON is a different problem from malformed JSON, because the part that exists is correct and the rest is missing. This article shows how to recognize a cut, what the parsers say, which repairs work and which only look like they do, and why the best repair is usually another request.

Recognizing a cut

A truncated document has a few signs. The last character is not } or ] (or a closing quote or digit that happens to end a top-level scalar). The parser's error position is the end of the text, or one step before it. The byte count is below what a Content-Length header or a manifest promised. And the text looks right up to the last few characters, which is the opposite of a syntax error in the middle.

The messages differ by what was cut. Node 24 said Unexpected end of JSON input when the text ended after a comma, and Expected ',' or '}' after property value in JSON at position 19 when it ended after a value in an unclosed object. Python 3.13 on the 60-character cut ending with a comma said Expecting value: line 1 column 61 (char 60), which is the position just past the end of the text. A cut inside a string produces Unterminated string starting at, with the position of the string's opening quote. When the error position equals len(text), or is within one of it, suspect truncation before anything else, and cross-check against the cases in unexpected end of JSON input.

Why the text is cut

Knowing the cause shapes the fix:

• A network or proxy timeout that closes the connection after part of the body was sent. The HTTP client may or may not raise; with Content-Length mismatch most clients do, but with chunked transfer a premature close can look like a normal end.

• A disk that filled up while writing a file, or a process killed during the write. Writing to a temporary name and renaming at the end prevents readers from ever seeing the partial file.

• A size limit in a log shipper, a database column or a message queue that truncates a long value. A MySQL TEXT column holds 65,535 bytes, and a JSON document longer than that, stored in a TEXT column in non-strict SQL mode, is cut silently on older configurations.

• A character limit in an interface, such as a chat message, a spreadsheet cell (Excel allows 32,767 characters) or a log line. The cut lands wherever the limit does.

• A generator that stopped, for example a language model that reached its output limit mid-document, or a streaming API that ended on error.

Each of these is a reason not to trust the text as the whole document, and each has a fix at the source that beats any repair.

Check the size before you debug

Do the cheap checks first. For HTTP, compare the Content-Length header with len(response.content); with requests you can compare them after the fact, and with curl -sI URL you see the header. For files, compare the size to a previous run, or to what the producer recorded. Then look at the tail: tail -c 100 file.json shows where it stops. A document that ends in the middle of a key is cut, and no amount of reading the middle will change that. If the producer supports it, ask for a checksum or a record count alongside the data so the receiver can verify completeness.

Salvaging complete records

If the document is a large array or a list of records, the first N records may be complete and perfectly usable. Two methods get them out without guessing.

A streaming parser. ijson reads incrementally and yields each item as it completes. On the test sample, cut at 98 of 173 characters, it produced items 0, 1 and 2, then raised IncompleteJSONError: parse error: premature EOF:

import io
import ijson

got = []
try:
    for item in ijson.items(io.BytesIO(cut.encode()), "items.item"):
        got.append(item)
except ijson.JSONError as e:
    print(type(e).__name__, e)
print(len(got), got[-1])

The printed result was 3 {'id': 2, 'name': 'n2'}. Catching the error and keeping what you have is the point: the items before the cut are correct, because each was fully parsed before it was yielded. The prefix items.item selects the elements of the array under the items key.

A decoder loop. Without extra packages, json.JSONDecoder().raw_decode reads one value at a time from a position. Starting just after the opening [, skip separators, decode, and stop at the first failure:

import json

def salvage(text):
    dec = json.JSONDecoder()
    pos, items = text.index("[") + 1, []
    while True:
        while pos < len(text) and text[pos] in " ,\n":
            pos += 1
        try:
            obj, pos = dec.raw_decode(text, pos)
        except json.JSONDecodeError:
            break
        items.append(obj)
    return items

It returned the same three items. It is a fine tool for a log excerpt or a saved response, and it is not a general parser, since it assumes the array of interest is the first one. The principle behind it, reading values one at a time from a position, is covered in Python JSONDecodeError extra data and concatenated objects. For big files use ijson, as described in handling large JSON files in Python with ijson.

JSON Lines. For one record per line, the cut leaves one bad line at most, the last. I cut a five-line file six characters before the end: four lines parsed and the last one, {"id, failed. Dropping a final line that does not parse is safe, as long as you note that you did it and compare the count with what you expected. This is a quiet advantage of the format, discussed in JSON Lines for data pipelines and log files.

Why closing the brackets is a trap

The tempting repair is to append whatever closers are missing. A tracker of open braces and brackets, aware of strings and escapes, can produce the closing sequence, and I wrote one. It works when the cut falls after a complete value. It failed on the other cuts:

• Cut after a comma, [... {"id": 1, "name": "n1"},, closing gave ...,]}, and Python 3.13 rejected it with Illegal trailing comma before end of array.

• Cut inside a key, {"id": 1, "nam, closing gave {"id": 1, "nam"}, which fails with Expecting ':' delimiter.

• Cut inside a value, "name": "n closes to "name": "n", which parses and is wrong, a truncated name stored as if complete.

A fix for each case is possible, and each one makes the code longer and still cannot recover the missing data. The last case is the dangerous one, because the repaired document is valid. Downstream code sees a record with a short name and a missing neighbor, and nothing flags it. The repair invented a smaller document and presented it as the original.

If you do close brackets, close at the last complete value, not at the cut: drop the partial tail first, and say in the output that the data is partial.

Preventing the cut

On the write side, write to a temporary file in the same directory and rename it when finished (os.replace in Python is atomic on the same filesystem), so a reader sees either the old file or the whole new one. Flush and check for write errors, since a full disk raises on close as well as on write. On the transport side, send a length or a checksum, use HTTP with Content-Length and verify it, and treat a connection reset as a failure, not as end of data. On the format side, JSON Lines with a trailing record count, or an envelope with "complete": true as the last key, gives the reader a way to know it has everything. For stored text, make the column big enough, and use the JSON type that validates on insert, which turns a silent truncation into a rejected row (see validating JSON before it gets into a MySQL column).

When repair is the wrong answer

Repairing a cut document is a recovery tool. These are the cases where it is the wrong tool.

When you can ask again. If the source is an API or a file you can regenerate, re-request it. A second attempt costs seconds and returns the true data, and a repair returns a guess. Add a retry with a check on the length so a cut is treated as a failure.

When the data drives money, access or compliance. A salvaged half of a ledger or an audit file is a different data set. Fail loudly and have a person decide.

When the cut is a symptom that will recur. A proxy that truncates large responses will truncate the next one. Fixing the limit is better than adding salvage to every consumer.

When the generator is a language model. A structured output that stopped at a token limit is incomplete. Raise the limit, shorten the request or use a format designed for streaming, such as JSON Lines, so that each completed record is usable by itself.

The specification of what makes a text complete is RFC 8259, and the streaming parser used above is documented on the ijson project page. For HTTP's rules on message length, see RFC 9112, section 6.3.