Premature End of Data in Tag and What Usually Caused It

You have a file that was fine yesterday, or that works on the producer's machine, and your parser stops with:

Premature end of data in tag orders line 2

The message is accurate and a little cryptic. The parser reached the end of the input while it was still inside an element that had not been closed. In practical terms, the file is shorter than it should be. The questions that matter are why, and what to do about the data that did arrive.

I cut a 28.6 MB file of orders in six ways and parsed each with ElementTree and lxml, then did the same with a gzip stream.

The ways a file can end early

Where the file stops decides which message you get. For the same source, I tried five cut points.

cut                       ElementTree                              lxml
between two records       no element found: line 252723, column 0  Premature end of data in tag orders line 2
inside a start tag        unclosed token: line 252723, column 2    Couldn't find end of Start Tag order line 252723
inside text               no element found: line 252724, column 19 Premature end of data in tag customer line 252724
10 bytes before the end   no element found: line 1200003, column 0 Premature end of data in tag orders line 2
inside a multibyte char   no element found: line 1, column 4       Premature end of data in tag a line 1

The ElementTree messages are about the position (nothing more to read, or a token not finished). lxml and xmllint name the element that was still open: orders, the root, in the cases where the cut came between whole records or near the end, and customer where it came in the middle of a value. The line 2 in "tag orders line 2" is where <orders> was opened. It is the root's own line, not the line where the file stopped; the line of the cut is the second number (line 252723).

That detail helps: if the open element is the root, the file was cut at a clean place or near the end and the whole body is probably fine. If the open element is something deep, the cut was in the middle of a record. In both cases the file did not arrive complete.

A multibyte case deserves a mention: a file cut in the middle of a UTF-8 character such as é gave the same "premature end" message and not an encoding error, because the parser had nothing more to decode.

Empty and nearly empty bodies

The degenerate case is a file with nothing in it:

(empty)         ET: no element found: line 1, column 0    lxml: Document is empty
"  \n"          ET: no element found: line 2, column 0    lxml: Start tag expected, '<' not found

A zero-byte file is what you get from a download that failed before it started, a command that redirected an error to a file, or a service that returned 204 No Content. Whitespace only is typical of a proxy or an upstream that answers with a newline. Before you debug the parser, check the size: ls -l file.xml or len(response.content).

Where truncation comes from

In practice, a few causes account for most cases:

An interrupted transfer. A download cut by a timeout, a dropped connection or a full disk. Check the exit status of the tool that fetched it, the HTTP status, and whether Content-Length equals the bytes you received. curl -f --retry 3 -o out.xml URL fails loudly on HTTP errors, and comparing wc -c out.xml with the Content-Length header shows a short body.

A producer that is still writing. A job that writes straight to the final name can be read while it is half done. The safe pattern for producers is to write to a temporary name and rename when finished, which is atomic on the same file system. For consumers: do not read a file whose modification time is within the last few seconds, or wait for a marker file.

A compressed stream that was cut. A truncated .xml.gz is a common variant. The gzip module is clear about it:

gzip.decompress(cut)
# EOFError: Compressed file ended before the end-of-stream marker was reached

Decompressing incrementally returned the part that was there (2,699,491 bytes from a stream cut at 90 percent of a 3,000,000-byte document) with eof false. Parsing that partial text failed at line 113956 with no element found. A truncated gzip always signals a truncated transfer, since the format has a checksum and an end marker.

A limit. A service that truncates large responses, a mail gateway that cuts attachments, a text field in a database (VARCHAR(65535) holding a document that is longer) or a log line limit. If the cut is always at the same size, look for the limit. A file of exactly 65,536 bytes, or 1,048,576, is a suspect.

A crash. The writer died halfway. Check its logs for an exception at the time the file was written, and its exit code.

What to do with the part that arrived

A partial document sometimes has value, and there are two safe ways to use it. iterparse processes events as it reads, so records that were complete can be handled before the error:

count = 0
try:
    for event, el in ET.iterparse(f, events=("end",)):
        if el.tag == "order":
            count += 1
            el.clear()
except ET.ParseError as e:
    print(f"read {count} complete orders before: {e}")
# read 42120 complete orders before: unclosed token: line 252723, column 2

42,120 complete orders were handled. If the processing is idempotent you can run it on the part and fetch the rest later.

The other way is lxml's recovery mode:

etree.fromstring(data, etree.XMLParser(recover=True))

On a copy cut mid tag it returned a root with 42,121 children and raised nothing. One more than iterparse counted, because the last order was incomplete: it exists in the tree with whatever fields were read. This is the danger of recovery. The result looks like a valid document, the count looks plausible, and the last record is damaged. Use recovery to inspect, not to load; if you must load, drop the last element and log that the file was truncated and how many records were lost.

A short checklist

1. ls -l the file and compare with the expected size. If you cannot know it, compare with yesterday's.

2. tail -c 200 file.xml. A complete file ends with the root's closing tag and a newline. Anything else is the cut.

3. Check the last HTTP status and Content-Length, and the exit code of the downloader.

4. For gzip, run gzip -t file.gz; it reports unexpected end of file.

5. Ask the producer whether it finished. Look for a marker or a checksum file.

6. Fetch again. Most truncations are transient.

7. Add a guard to your pipeline: parse the whole file before you load any of it, or stream and commit only if the parser reaches the end without error. A load that commits record by record leaves a half-loaded batch behind when the parse fails.

The sibling article on locating errors in a big file shows how to read the position the parser gives (XML parsing error not well formed and how to find the spot), and fixing XML parse errors fast puts the whole set of checks on one page. For files too big to hold, parsing large XML files with iterparse is the safe way to read in pieces. The libxml2 documentation describes the error in its error code reference, and the Python gzip module is documented at docs.python.org. To look at the end of a small file in the browser, the XML formatter shows where the structure stops.

When the file is complete and still fails

The closing tag is there but misspelled. That is a mismatch, not a premature end, and the message says so.

An entity or CDATA section is unterminated. <![CDATA[ without ]]> makes the parser read to the end looking for it. The error names the CDATA or reports the end of input.

The end of the file is fine and the beginning is wrong. A fragment that was pasted without its root, or a second document appended to the first, gives different messages (extra content, start tag expected).

You are reading from a stream that is not finished. A socket or a pipe that you read once yields the bytes available so far. Read until end of stream before you parse.