Mismatched Tag Errors in Large XML Files and Finding Them

In a short document, a mismatched tag is easy to see. In a file of 200,000 records, 1.2 million lines and 28.6 MB, it is one wrong word among 5 million. The parser knows where, and usually that is the end of the problem. There are cases where it is not: the report is far from the cause, there are several errors, or the message is unclear. This article covers four ways to locate a mismatch, from the obvious to the fallback, with measured times.

What the error means

Opening and ending tag mismatch: status line 900005 and state

The parser opened <status> on line 900005 and the next closing tag it read was </state>. Closing tags must name the element that is currently open, so this is an error. There are three ways it arises:

A typo in a closing tag (state for status), as here. A forgotten closing tag earlier, so the parser sees a closing tag for the parent while a child is still open. Crossed nesting, <b><i>text</b></i>, common in text that was hand edited or converted from HTML.

The message names both the tag that was open and where it was opened, and the line where the wrong closing tag was found.

Method 1: let the parser say

Three tools, same file:

xmllint --noout           0.33 s   Opening and ending tag mismatch: status line 900005 and state
xmllint --stream --noout  (not timed)   same message, no tree in memory
lxml etree.parse          0.30 s   ... line 900005, column 24
ElementTree.parse         0.48 s   mismatched tag: line 900005, column 17

For a typo this is the whole answer: go to the line, fix the tag. xmllint prints the line with a caret, so you can read the context without opening the file. --stream avoids loading the tree, which matters on files larger than memory. The first message is the only one to trust; later ones are consequences (XML parsing error not well formed and how to find the spot).

Method 2: count tags by name

When the parser's message is far from the cause, counting helps. The idea: in a well-formed file, every tag name is opened as many times as it is closed.

for tag in (b"status", b"order", b"customer"):
    opens  = data.count(b"<" + tag + b">") + data.count(b"<" + tag + b" ")
    closes = data.count(b"</" + tag + b">")
    print(tag.decode(), opens, closes)

status    200000  199999
order     200000  200000
customer  200000  200000

One name is unbalanced: status was opened 200,000 times and closed 199,999. That tells you which element to look for: a missing or misnamed closing tag for status, somewhere. It does not say where. The count is also fast (milliseconds for a million lines with grep -c).

A cruder version of the count fails, and the failure is worth knowing. A running depth (add one for every opening tag, subtract one for every closing tag) ended at 0, and never went negative. The wrong name </state> is still one closing tag, so the total depth balances; only the names disagree. A depth count finds missing or extra tags and cannot find wrong ones. Counts by name catch both, as long as only one name is affected.

Method 3: parse each record on its own

If the file is a sequence of similar records, cut it into records and parse each one separately. The failing record is the one that raises:

for m in re.finditer(rb'<order id="(\d+)">.*?\n  </order>', data, re.S):
    try:
        etree.fromstring(m.group(0))
    except etree.XMLSyntaxError as ex:
        print(m.group(1), ex)

This ran over all 200,000 records in 0.72 s and found one bad record, id 150000, with the message Opening and ending tag mismatch: status line 3 and state. Line 3 is relative to the record. The regular expression relies on the file's layout (each record ends with </order> on its own line), which is a weakness. If the breakage affects the record boundary itself, the regex will swallow the next record or fail to match. For files that are not neatly laid out, this method needs a more careful splitter.

Its strength is that it finds every bad record and not just the first. If the file has fifty damaged records, you get fifty answers, where the parser stops at one. Good records can be loaded, and bad ones written to a reject file with their ids.

Method 4: bisect on well-formed prefixes

The fallback that works with no assumptions about the content is to bisect. Take the first k records, add the closing root tag, and test whether that parses. If it does, the problem is later; if not, it is earlier.

lo, hi = 0, len(records)
while lo < hi:
    mid = (lo + hi + 1) // 2
    try:
        etree.fromstring(data[:record_end[mid - 1]] + b"</orders>")
        lo = mid
    except etree.XMLSyntaxError:
        hi = mid - 1
# lo is the last good record

On 200,000 records it took 17 parses and 4.34 s and reported 150,000 as the last good record, so record 150,001 (the one with id 150000) is the first bad one. The cost is a parse of a large prefix each step, which makes it slower than the others on this file. Its virtue is that it needs only a function that tells good from bad, so it also works when the parser's message is unhelpful, and when the "good or bad" test is something else, such as a schema validation or a load into a database. The same trick finds the one row that breaks an import.

What to do with a file you cannot fix

If the source cannot send a corrected file, you can still use the good part. iterparse streams the elements and tells you how many records were complete before the failure:

for event, el in ET.iterparse("mismatch.xml", events=("end",)):
    if el.tag == "order":
        count += 1
        el.clear()
# 150000 complete orders before: mismatched tag: line 900005, column 17

150,000 complete orders in 0.55 s. Process them, then repair a copy (fix the tag in place with a one-line edit once you know the line) and read the rest. Keep the original, and write down what you changed.

lxml's recover=True will parse past the error, and I would not use it for a mismatch: it must guess which tag was meant, and the guess changes the tree (in the earlier test it merged a record into its neighbor). It is a way to look at a file, not to load it.

Keeping it from happening

Mismatched tags in a large file nearly always come from string concatenation or a template, an interrupted writer, or hand editing. Generate XML with a serializer, which cannot produce a mismatch. Validate the file when it is produced and again when it is received. For a producer that writes a record at a time, write a record to a buffer, parse it to check, and then append it. Add a checksum or a record count to the transfer so truncation and corruption are caught at the boundary (premature end of data in tag). More on the whole list of errors is in fixing XML parse errors fast, and for reading large files without loading them, parsing large XML files with iterparse.

The parser behavior is documented in the xmllint manual and the Python iterparse reference. To check a short excerpt, the XML formatter shows where a snippet's structure goes wrong.

When the parser's answer is enough

There is one error and it is a typo. The message and the line are the answer. Fix it and move on.

The file is small. Under a few thousand lines, an editor with tag matching finds the problem faster than any script.

You can regenerate. If the producer is under your control, the bug is in the generator.

There are many errors. Then method 3 is worth the effort: one bad record in a million is a problem to repair, and ten thousand is a sign that the file was produced wrongly.