Fixing XML Parse Errors Fast With a Triage Routine

An XML parse error is the kind of failure that looks like a puzzle and is mostly a checklist. The parser reports one problem at one position. A few causes account for nearly every case, and they can be checked in a fixed order, cheapest first. This guide is the routine, the message dictionary that goes with it, and a small script that runs the first half of the routine for you.

It pulls together the individual articles in this cluster. Each of them has the experiment for one cause; this one has the order in which to look.

Why a fixed order

When a file fails, the temptation is to open it and look. For small files that works. For anything else it fails, because the cause is often invisible: a byte order mark, a vertical tab, a Latin-1 accent in a file that says UTF-8. The parser also reports only the first problem, and sometimes reports it in terms of its own state ("not well-formed (invalid token)") and not in terms of the cause.

The routine reverses the usual order. Instead of asking the parser first, ask cheap questions of the bytes first, because their answers are certain, and use the parser last, for what is left.

1. Is it XML at all?                 first bytes
2. Is it the file I think it is?     size, end of file, compression
3. Is the encoding right?            declaration vs. bytes
4. Are the characters legal?         control characters, ampersands
5. Does the structure hold?          the parser's message and position

Steps 1 to 4 take seconds on any file size and name the cause in most cases. Step 5 is where the specific articles come in.

Step 1: confirm it is XML at all

Look at the first bytes: head -c 200 file | xxd | head. The healthy start is < (0x3C), optionally after a UTF-8 BOM (EF BB BF). Anything else tells you something.

{ or [          JSON (an API returned an error body in JSON)
a letter        plain text (a service message, a stack trace)
1F 8B           gzip (the response was compressed and not decoded)
FF FE or FE FF  UTF-16 byte order mark
<!DOCTYPE html  an HTML page (a login redirect, a 404)
spaces, newline whitespace before <?xml

In my tests the JSON body, the text message and the gzip bytes were all rejected by Java with Content is not allowed in prolog or Invalid byte 1 of 1-byte UTF-8 sequence, by lxml with Start tag expected, '<' not found, and by ElementTree with an invalid token or a syntax error at line 1, column 0. Every parser says the same in different words: the first thing in the file is not a tag (content is not allowed in prolog covers the Java case and the hidden BOM). Never debug the parser before you have looked at what the server sent. Print the status code, the content type, and the first 200 bytes.

One warning from the same tests: an HTML page, <!DOCTYPE html><html>..., parsed without error as XML because it was well formed. A successful parse does not prove it is the document you wanted.

Step 2: confirm it is complete

Look at the size, the end of the file, and the transfer. A complete document ends with the closing root tag. tail -c 100 file | xxd shows what the file ends with. In the truncated test the tail was ...</status>\n <to, which is a cut inside a tag. The parsers said Premature end of data in tag orders line 2 (lxml), unclosed token or no element found (ElementTree), and Couldn't find end of Start Tag (xmllint), each at the last line of the file (premature end of data in tag).

If the file is smaller than expected, the cause is upstream: a failed or interrupted transfer, a writer that is still running, a truncated gzip (EOFError: Compressed file ended before the end-of-stream marker was reached), or a size limit. Fetch again and compare sizes before anything else.

Step 3: check the encoding

Read the declaration (head -1 file) and compare it with what the bytes are. file file.xml reports the bytes. The checks that matter:

Strict UTF-8 decoding of the bytes. If the declaration says UTF-8 (or there is none) and the bytes fail to decode, the file is not UTF-8. The error says where: Input is not proper UTF-8, indicate encoding ! Bytes: 0xE9 0x20 0x5A 0x6F. Latin-1 or Windows-1252 is the usual answer for Western text.

A declaration that disagrees with valid UTF-8 bytes. This one produces no error. A UTF-8 file declared as ISO-8859-1 parsed to café in every tool. Look for Ã in the output.

A BOM and a declaration that disagree, which parsers handle differently (XML encoding declarations and the wrong encoding problem).

Encoding detection libraries guess well for UTF-8 and UTF-16 and badly for Latin-1 and cp1252: on Latin-1 French and German samples they answered cp1250 and MacCyrillic, so use them as a hint and not a verdict (encoding mismatch errors in XML documents has the table and the UTF-8-first rule).

Step 4: check the characters

Two scans over the bytes find the commonest character problems, and each finds all instances, where the parser stops at the first.

Forbidden control characters. [\x00-\x08\x0b\x0c\x0e-\x1f]. XML 1.0 forbids them raw and as character references. In the 28.6 MB test file one vertical tab was found by a scan in 0.09 s and removed in 0.03 s with bytes.translate (invalid character in XML and stripping control characters).

Ampersands that do not start a legal reference. &(?!(?:amp|lt|gt|quot|apos|#[0-9]+|#x[0-9a-fA-F]+);). In the legacy export it found 215,385 of them. The same pattern is the repair, with & as the replacement, and it is safe to run twice, unlike a plain replace of & that double-escapes the entities that were right (fixing XML with unescaped ampersands from a legacy export). HTML entity names such as   and é fall in the same group but need conversion to numeric references, not escaping (undefined entity errors in XML).

A script that runs steps 1 to 4

The first four steps are mechanical, so I wrote them down as a script, about 60 lines, xmlcheck.py. It reads the bytes, checks the first character and the BOM, compares the declared encoding with strict UTF-8 decoding, counts forbidden control bytes and illegal ampersands, looks at the end of the file, and only then runs the parser and appends its message.

def triage(path):
    b = open(path, "rb").read()
    notes = []
    if not b:                      return "empty file (0 bytes)"
    if b[:2] == b"\x1f\x8b":       notes.append("gzip data: decompress first")
    first = b.lstrip(b"\xef\xbb\xbf").lstrip(b" \t\r\n")
    if first[:1] != b"<" and not notes:
        return f"not XML: first byte is {first[:1]!r}: {first[:60]!r}"
    ...
    ctrl = re.findall(rb"[\x00-\x08\x0b\x0c\x0e-\x1f]", b)
    amp  = len(re.findall(rb"&(?!(?:amp|lt|gt|quot|apos|#[0-9]+|#x[0-9a-fA-F]+);)", b))
    ...
    try: etree.parse(path); notes.append("parses OK")
    except etree.XMLSyntaxError as e: notes.append(f"parser: {str(e).split(',')[0]} (line {e.lineno})")
    return "; ".join(notes)

I ran it over 18 files, 12 of which fail to parse. A selection of the output:

prolog/json.xml:  not XML: first byte is b'{' (JSON): b'{"error":"unauthorized"}'
prolog/text.xml:  not XML: first byte is b'E' (text/other): b'Error: service unavailable'
prolog/space.xml: whitespace before the XML declaration; parser: XML declaration allowed only at the start of the document (line 2)
enc/latin1_decl_utf8.xml: bytes are not valid UTF-8 at offset 44, but declared UTF-8; parser I/O: Error reading file ... Invalid bytes
big/ctrl.xml:     1 forbidden control byte(s), e.g. b'\x0b'; parser: PCDATA invalid Char value 11 (line 420004)
big/truncated.xml: does not end with '>' (tail b'aid</status>\n    <to'): truncated?; parser: Couldn't find end of Start Tag to line 745770
big/legacy.xml:   215385 ampersand(s) not part of a legal reference; parser: xmlParseEntityRef: no name (line 4)
big/mismatch.xml: parser: Opening and ending tag mismatch: status line 900005 and state (line 900005)

For the byte-level problems the script named the cause on its own line before the parser said anything useful. For the structural one (the mismatch) it had nothing to add and the parser's own message was the answer, which is the intended division of labor.

It has false alarms, and they are worth knowing. A UTF-16 file was flagged "does not end with >" because its last byte is a null; a gzip file produced a cascade of notes after the first (the first line, "gzip data: decompress first", was the right one). It also cannot see the silent encoding mistakes: utf8_decl_latin1.xml and cp1252_curly.xml both printed "parses OK". Treat the script as a first look that points at the likely cause, and read its first note.

Step 5: the parser's message

If the first four steps found nothing, the cause is structural, and the parser's message maps to a short list:

The table is the compressed version of the cheat sheet of syntax rules (a cheat sheet of XML syntax rules), which lists 51 snippets with their messages. For the unbound prefix case see Python ParseError unbound prefix in XML.

Three failures worked end to end

The routine is easier to trust after seeing it run on real failures. Three from the test set, with the steps in the order I took them.

The export that broke every night

A legacy system's export is rejected by the loader with not well-formed (invalid token): line 4, column 19. Step 1 is clean: the file starts with <?xml. Step 2: it is 29.2 MB, the size of the last few runs, and it ends with </orders>. Step 3: the declaration says UTF-8 and the bytes decode as UTF-8. Step 4 finds the cause at once: 215,385 ampersands that do not start a legal reference, and no control characters. The parser agrees (xmlParseEntityRef: no name at line 4), and says nothing about the other 215,384.

The decision here is between repairing and rejecting. The ampersands are customer names (Tom & Jerry) and notes (R&D), so the content is fine and the escaping is missing. A lookahead regex escapes exactly the bad ones in 0.08 s, the repaired file parses into 200,000 orders in 0.77 s, and the loader runs. The producer is told, and a counter goes in the log. The next night the count is 215,401, which is a number worth having when the conversation with the sender starts. Total time from alert to loaded data: a few minutes, because the routine pointed straight at the cause.

The file that was fine on the sender's machine

A partner's file fails with Input is not proper UTF-8, indicate encoding ! Bytes: 0xE9 0x20 0x5A 0x6F. Step 1 and 2 pass. Step 3 is where it ends: the declaration is UTF-8 and the bytes E9 20 5A 6F are é Zo in Latin-1. file reports ISO-8859 text. The sender's tools were lenient about it (some parsers fall back, some do not). Two ways to proceed. Convert the bytes with iconv -f ISO-8859-1 -t UTF-8 and fix the declaration on a copy; or tell the parser the true encoding (XMLParser(encoding="iso-8859-1") in lxml). Both gave café Zoë. If the text had curly quotes, the encoding would be Windows-1252, not Latin-1, and the conversion must say so, or the quotes come out as control characters with no error.

The point of the routine here is that the parser's message does not say "wrong encoding". It says invalid UTF-8, and only Step 3 turns it into the right question.

The download that stopped halfway

A nightly job reports Couldn't find end of Start Tag to line 745770 from xmllint and unclosed token: line 745770, column 4 from ElementTree. The line number is far into the file but not at the end of what the file should have been. Step 2 shows the problem: the tail is ...</status>\n <to and the file is 17.7 MB where yesterday's was 28.6 MB. It was cut in the middle of a tag. The parser's message is accurate and does not mention truncation. Nothing in the file is wrong before the cut: iterparse counted 42,120 complete orders before it stopped at the same place.

The fix is a new download, and a guard: check the size against the expected one, check the HTTP Content-Length, and fail the job before it loads a partial batch. If the loader had committed orders as it parsed, the batch would have been a tenth loaded, and every later run would have wondered about duplicates.

A quarantine loader

Once the routine is clear, a pipeline can embody it. The structure that has held up:

def ingest(path):
    findings = triage(path)                      # steps 1-4, byte level
    if findings.fatal:                           # not XML, empty, gzip, truncated
        quarantine(path, findings); return
    try:
        tree = parse_untrusted(path)             # defusedxml or a locked-down lxml parser
    except ParseError as e:
        quarantine(path, findings + [str(e)]); return
    errors = validate(tree, schema)              # structure and types, if a schema exists
    if errors:
        quarantine(path, errors); return
    load(tree)                                   # only now
    archive(path)

Three properties matter. Nothing is loaded until the whole file has been parsed and validated, so a failure never leaves half a batch. A failing file is moved to a quarantine directory with its findings in a file beside it, so whoever investigates starts with the diagnosis. And the original bytes are archived after success, so a later bug in the loader can be fixed by replaying the file.

A refinement for large feeds: parse in a streaming fashion and load records into a staging table, then swap the table in if the whole parse succeeded. That keeps memory flat and still gives an all-or-nothing result.

Reading the symptoms

A compressed version of the routine, by what you see.

If the error is on line 1, column 1 or 0, suspect Step 1: it is not XML, or there is a BOM or whitespace before the declaration.

If the error is at the end of the file, or on the last line, suspect Step 2: truncation.

If the error mentions bytes, UTF-8, or invalid sequences, suspect Step 3.

If the error mentions a character value, an entity or an ampersand, suspect Step 4.

If the error names two tags, an attribute or a prefix, it is structural, and the position is close to the cause. For a mismatch, read upward from the position.

If the file parses and the text looks wrong (Ã, \x93, missing letters), the problem is the encoding, and no parser will tell you.

How the messages differ between tools

The same cause is worded differently by each parser, which is why a search for the exact text sometimes returns the wrong article. For a raw ampersand, xmllint and lxml say xmlParseEntityRef: no name and ElementTree says not well-formed (invalid token). For a closing tag that does not match, xmllint and lxml name both tags and ElementTree says mismatched tag. For text before the root, lxml says Start tag expected, '<' not found, ElementTree says syntax error: line 1, column 0 or an invalid token, and Java says Content is not allowed in prolog. When a message is vague, run the file through xmllint --noout, which is the most descriptive of the three and installs everywhere libxml2 does. When a message from one tool does not match any article, translate it into the underlying cause (a tag, an entity, a character, an encoding, an ending, a start) and look up that cause.

Rules that save time

Fix the first error only. A parser stops at the first problem, and a tool with recovery reports follow-on errors that vanish when the first is fixed. Run again after each fix.

Do not use recovery to load data. lxml's recover=True and xmllint --recover returned trees that silently dropped undefined entities, merged records and kept a damaged last element (42,121 orders recovered from a file cut in the middle of the 42,121st). Use recovery to look, never to load.

Keep the original untouched. Repair a copy, count what you changed, and keep the count in a log.

Use the right tool for the job. xmllint --noout for a verdict and a caret under the error; --stream for files too large to build a tree; lxml's error log for every message in a script; iterparse to see how many records were fine; the indenting tokenizer to see the shape of a structural mistake in a minified file (pretty printing XML to spot structural mistakes).

Locate before you fix. On a 28.6 MB file the parsers found the error in 0.2 to 0.5 s (XML parsing error not well formed and how to find the spot). Looking at the file by eye is the slow way.

Fix at the source. The pipeline is cleaner if the producer stops making the mistake. An export that needs 215,385 repairs this week will need them next week, and the sender is the only one who can end that.

Why messages mislead

Parser messages describe the parser's state, not your mistake, and a few patterns recur. Knowing them cuts the time to a fix.

The position is where the parser noticed. For a missing closing tag the parser reports the line of the wrong closing tag, which can be dozens of lines below the real mistake. For a truncated file it reports the last line, whatever caused the cut. For an encoding error it reports the first bad byte, which may be far from the top of a file that begins with plain ASCII.

The wording changes with the tool. The same raw ampersand is xmlParseEntityRef: no name in libxml2 and not well-formed (invalid token) in expat, and the same stray text before the root is Start tag expected, '<' not found, syntax error: line 1, column 0 or Content is not allowed in prolog. Searching for the exact words of one tool finds articles about another's messages.

Errors cascade. After the first mismatch, every closing tag looks wrong. Of the four messages xmllint printed for a minified file with one bad </items>, only the first was real. Read the first, fix it, run again.

A successful parse is weak evidence. A well-formed HTML error page, a UTF-8 file declared as Latin-1 and Windows-1252 text declared as Latin-1 all parsed without complaint in my tests. When the output looks wrong and the parser is happy, check the root element's name and look for Ã and control characters in the text.

What to measure in a feed

The same checks that diagnose a failure also describe the health of a feed over time. Track a handful of numbers per file and plot them. File size, because a drop of half is a truncation or an outage upstream. Number of top-level records, for the same reason. Count of repaired ampersands and removed control characters, because a rise means the content changed. Declared versus detected encoding, which should never disagree. Parse time per megabyte, which is stable until the document changes shape.

On the 28.6 MB test file, a full parse took 0.2 to 0.7 s depending on the library, a byte-level scan for control characters took 0.09 s, and the ampersand pattern 0.08 s. Running all the cheap checks on every file adds well under a second per 30 MB, which is much cheaper than finding out from a failed load.

Quarantine is the other half. Keep failed files, with their triage output, for long enough that the sender can be asked about them. A folder of fifty failures with the same message is a pattern, and a pattern is something a sender can fix once.

What not to do

A few habits make parse errors worse.

Do not edit the file by hand in a text editor that rewrites encodings or line endings. Saving a UTF-8 file with a BOM, or converting CR LF, changes bytes you did not mean to change, and the fix creates a second problem. Make changes with a script and keep the script.

Do not swallow the error. A try/except that logs "bad file" and moves on turns a loud failure into missing data. Catch the error to quarantine the file and alert.

Do not parse with recovery on in production. It returns something, and what it returns is a guess.

Do not "fix" encodings by trial and error on the text itself. Replacing é with é by string substitution works for one character and leaves the rest of the document damaged. Convert the bytes correctly, once, from the right source encoding.

Do not run an untrusted file through a parser with entity resolution enabled while you debug. The same tools that read your own files can read your filesystem when a document says so (what a DTD is and why you rarely need one).

A last look at the numbers

To close the loop, here is how the cheap checks compare with the parser across the large-file cases, on 28.6 MB.

case                    cheap check                         time      parser                     time
control character       forbidden-byte scan, 1 found        0.09 s    lxml PCDATA invalid Char   0.23 s
raw ampersands          lookahead scan, 215,385 found       0.08 s    lxml xmlParseEntityRef     0.24 s
truncated file          tail does not end with '>'          0.01 s    lxml Couldn't find end     0.17 s
mismatched tag          (none: structural)                  -         lxml tag mismatch          0.30 s
wrong encoding          UTF-8 strict decode fails           0.01 s    lxml Invalid bytes         not timed

The parser is fast enough on these sizes that speed is not the argument for the routine. The argument is information. The scans find every instance, where the parser finds the first, and they name the cause in words you can act on. The parser, in turn, is the only tool that sees structure. Use them in that order and neither is wasted.

The rest of the cluster takes each cause in depth: XML parsing error not well formed and how to find the spot for reading positions, mismatched tag errors in large XML files for locating structure problems at scale, and encoding mismatch errors in XML documents for the cases detection cannot settle.

Making the routine permanent

Once you have run the routine twice, put it in the pipeline. Receive the file, run the byte checks, record the findings, parse, validate against the schema if there is one, and only then load. Keep the failures in a quarantine folder with the triage output beside each file, so whoever picks it up has the diagnosis. Count failures by cause per day: a rise in "not XML" points to an authentication or routing problem, a rise in "truncated" points to the transfer, a rise in raw ampersands points to a change in the content. Those counts tell you about the source system sooner than the sender's own monitoring does.

For validation after parsing, well formed versus valid XML explains the two levels, and for untrusted input the safe parser configuration is in what a DTD is and why you rarely need one. The tools are documented in the xmllint manual and the lxml parsing guide. For a quick look at a small document, use the XML formatter.

When the file is not the problem

The consumer is the problem. A parser configured with a schema, a resolver that cannot fetch a DTD, or a size limit can reject a perfectly good file. If other parsers accept it, look at the configuration.

The error is real and expected. A feed that validates input and rejects bad documents is working. The fix is a better report to the sender, not a more tolerant parser.

The file is fine and the transport changed it. Proxies that rewrite bodies, mail gateways that wrap lines and tools that "pretty print" can all alter a document in transit. Compare checksums at both ends.

The format is not XML. Some HTML-like or SGML files look like XML and are not. An HTML parser is the right tool.