XML Parsing Error Not Well Formed and How to Find the Spot

"Not well-formed (invalid token)" is the most common XML error message there is, and one of the least helpful. It tells you that the document broke a syntax rule and, if you are lucky, where. What you do next determines whether the fix takes a minute or an afternoon.

I broke a large file in a controlled way and watched what three parsers reported. Then I used the output to build a procedure for finding the problem in a file you cannot read by eye.

What the error says

A parse error has three parts: a message, a position and, in some tools, the offending line. For the changed closing tag (<status>new</state> on line 900005) the three tools said:

ElementTree   ParseError: mismatched tag: line 900005, column 17
lxml          XMLSyntaxError: Opening and ending tag mismatch: status line 900005 and state, line 900005, column 24
xmllint       big/mismatch.xml:900005: parser error : Opening and ending tag mismatch: status line 900005 and state
                  <status>new</state>
                                     ^

All three put the error on the right line in 0.2 to 0.5 s on 28.6 MB. The messages differ in how much they tell you. ElementTree says "mismatched tag" and gives a position. lxml and xmllint name both tags, so you know a status was opened and a state closed it. xmllint also prints the line and a caret under the character where it stopped, which is the quickest thing to read.

The same file with a raw ampersand in <customer>Smith & Sons</customer> (line 540010):

ElementTree   not well-formed (invalid token): line 540010, column 21
lxml          xmlParseEntityRef: no name, line 540010, column 22
xmllint       parser error : xmlParseEntityRef: no name
                  <customer>Smith & Sons</customer>
                                  ^

and with a vertical tab (code 11) in a name:

ElementTree   not well-formed (invalid token): line 420004, column 18
lxml          PCDATA invalid Char value 11, line 420004, column 19
xmllint       parser error : PCDATA invalid Char value 11

Look at what ElementTree said both times: the same words, "not well-formed (invalid token)", for an ampersand and for a control character. Expat reports a category of failure and a position, and libxml2 reports the specific rule. When you get the generic message, run the file through xmllint or lxml for a better description. They agree on the line (the column differs by one because of how each counts).

Getting the position in code

In ElementTree the exception carries a tuple:

try:
    ET.parse("orders.xml")
except ET.ParseError as e:
    line, col = e.position       # (540010, 21)
    code = e.code                # 4

lxml gives e.lineno and e.offset, and its parser keeps a log with every message, which matters when there is more than one problem:

try:
    etree.parse("orders.xml")
except etree.XMLSyntaxError as e:
    for entry in e.error_log:
        print(entry.line, entry.column, entry.message)

To show the surrounding lines, read the file and print a window around the reported line:

lines = open("orders.xml", encoding="utf-8").read().splitlines()
for n in range(line - 3, line + 2):
    print(n + 1, lines[n])

On the command line, sed -n '540007,540012p' file.xml does the same without opening an editor. Beware of single-line files: a minified document has one line, so the line number is always 1 and the column is the only guide. For those, xmllint --format on the pieces that parse, or the column number against cut -c, helps.

The method

For an XML file that does not parse, this sequence finds the problem most of the time.

1. Fix only the first error. A parser stops at the first well-formedness error. Later messages, in a tool that recovers, are often consequences of it. A single unclosed tag can produce a hundred errors below it.

2. Read the message category. There are about a dozen causes of "not well-formed" in practice. The messages map to them:

Opening and ending tag mismatch     a tag closed with a different name, or in the wrong order
xmlParseEntityRef: no name          a raw & (see the escaping article)
Unescaped '<' not allowed           a raw < in an attribute (or text)
AttValue: " or ' expected           an unquoted attribute value
Attribute x redefined               a repeated attribute on one element
PCDATA invalid Char value N         a control character that XML does not allow
Premature end of data in tag        the file ends inside an element (truncated)
Extra content at the end            a second root, or text after the root

The cheat sheet of syntax rules has each of them with an example.

3. Look at the reported line, and the one above. For a mismatch the parser reports where it found the wrong closing tag. The mistake might be an opening tag a few lines earlier that was never closed: the parser sees </order> while it expects </status>. Read upward until the nesting makes sense.

4. Check the character at the column. For an ampersand or a control character the caret points at it. For an "invalid token" it is usually the character just before or at the column. Print it with its code point (ord(line[col-1])) when it is invisible.

5. If the file is huge, do not open it in an editor. The command-line tools reported the problem in half a second on 28.6 MB. Use them first.

6. Fix at the source. If you generated the file, fix the generator, since every file it writes will have the same problem. If a partner sent it, tell them, and in the meantime repair a copy (fixing XML with unescaped ampersands from a legacy export and stripping control characters cover the two common repairs).

When the position is not the cause

The position is where the parser noticed the problem, and for some errors that is a long way from where the problem was made.

A missing closing tag is the standard example. If <status> is never closed, the parser keeps reading as if it were still inside, and only complains when it meets something that cannot belong there, perhaps at the end of the next </order>. The reported line is the one with the wrong closing tag; the mistake is above.

A truncated file reports an error at the end, whatever the mistake. ElementTree said unclosed token: line 745770, column 4 for a file I cut in the middle of a start tag, and xmllint said Couldn't find end of Start Tag to line 745770 followed by Premature end of data in tag order line 745767. The last line of the file is the problem, and the right question is why the file ends there (premature end of data in tag).

A wrong encoding reports an error at the first byte that does not decode, which may be far into a file whose first lines are plain ASCII (XML encoding declarations and the wrong encoding problem).

Recovering a partial document

Sometimes you need whatever can be read. lxml has a recovery mode that continues past errors:

r = etree.fromstring(data, etree.XMLParser(recover=True))

On a copy of the file cut mid-tag it returned a tree with 42,121 order elements (the ones before the cut) and no error. That is useful for diagnosis and for salvaging, and dangerous as a habit, because it also hides the problem. A pipeline that parses with recover=True and loads the result has silently dropped part of the data. Log the error log when recovery was needed, and compare the count of records with what you expected.

Streaming helps too. ET.iterparse processes elements as it reads, so you can count the records that came through before the failure (42,120 complete orders in that example), which tells you how much of the file is good.

Tools worth keeping close

xmllint --noout file.xml for the first answer. xmllint --stream --noout for a large file without building the tree (same message, same line). lxml with the error log for scripts. ElementTree's position for tools that already use it. A hex view (xxd | head) for the first bytes when you suspect a byte order mark or an encoding problem. And the XML formatter for a quick look at a small document in the browser. Related reading: fixing XML parse errors fast puts these into a one-page procedure. The parser behavior is described in the lxml documentation on parsing and the libxml2 error list.

When the reported spot is not the cause

Truncation. If the position is on the last line, the file was cut, and the fix is to get the whole file.

An earlier unclosed tag. Read upward from the report until the nesting is right.

Encoding. A byte error in the middle of a clean file means the bytes are not what the declaration says.

Generated markup. If a template produced the file, the bad spot is in the template or its data, and the line number points at the output and not at the code. Reproduce the failure with the smallest input that triggers it.