Pretty Printing XML to Spot Structural Mistakes Quickly

A well-formedness error tells you the line and column where the parser noticed a problem, and in a minified file that is "line 1, column 121". There is no line to look at. The instinct is to pretty print the document and read it. The trouble is that formatters need a document that parses, and a document that does not parse is exactly the one you cannot format.

I tried the tools that exist and then wrote a small one that works on broken input.

The problem file

<orders><order id="1"><customer>Ana</customer><items><item sku="A"><qty>2</qty></item><item sku="B"><qty>1</qty></items><total>31.0</total></order><order id="2">...

The second item is never closed: it ends with </items>, so the parser sees </items> while it expects </item>. On one line, with no indentation, that is hard to see.

What the formatters do

xmllint --format is the usual way to indent. Given the broken file it printed errors:

pp/min.xml:1: parser error : Opening and ending tag mismatch: item line 1 and items
pp/min.xml:1: parser error : Opening and ending tag mismatch: items line 1 and order
pp/min.xml:1: parser error : Opening and ending tag mismatch: order line 1 and orders
pp/min.xml:2: parser error : Premature end of data in tag orders line 1

No formatted output. The errors cascade: the first is real, and the next three are consequences, because once the stack is out of step every later closing tag looks wrong too. With --recover, the same four errors and then the document echoed back, still on one line, and changed:

<orders><order id="1">...<item sku="B"><qty>1</qty></item><total>31.0</total></items><order id="2">...</order></order>

The recovery guessed. It closed the open item at </items>, kept going, and wrapped the whole second order inside the first (note the </order></order> at the end). The structure it printed is not the structure that was written, so reading it would mislead you.

lxml with recover=True behaved the same. Its error log had three entries (columns 121, 148 and 268) and its tree had moved elements. pretty_print=True on that tree did not indent anything in this case: the recovered nodes carried no whitespace and the serializer left the single line as it was. Neither tool gives you a trustworthy indented view of a file that does not parse.

A tokenizer that does not need a valid document

What helps is a reader that is less ambitious than a parser. It only needs to split the text into tokens, track which tags are open, and print each token at the depth of the stack. About thirty lines of Python:

TOK = re.compile(r"<!--.*?-->|<!\[CDATA\[.*?\]\]>|<\?.*?\?>|<![^>]*>|</[^>]+>|<[^>]*?/>|<[^>]+>|[^<]+", re.S)

def show(s):
    stack, out = [], []
    for t in TOK.findall(s):
        if t.startswith("</"):                              # closing tag
            name = t[2:-1].strip()
            if stack and stack[-1] == name:
                stack.pop(); out.append("  " * len(stack) + t)
            else:
                out.append("  " * len(stack) + t + f"      <-- MISMATCH (open: {stack[-1] if stack else 'none'})")
                if name in stack:
                    while stack and stack.pop() != name: pass
        elif t.startswith("<") and not t.startswith(("<!", "<?")) and not t.endswith("/>"):
            out.append("  " * len(stack) + t); stack.append(re.match(r"<([^\s/>]+)", t).group(1))
        elif t.strip():
            out.append("  " * len(stack) + t.strip())
    if stack: out.append(f"<-- unclosed at end: {stack}")
    return "\n".join(out)

The regular expression lists the kinds of token in order of priority: comments, CDATA sections, processing instructions, declarations, closing tags, self-closing tags, opening tags, and then text. For each token the function prints it at the current depth. An opening tag pushes its name; a closing tag pops, but first checks that the name on top of the stack matches. If it does not, the line is marked with the open element and the stack unwinds to the matching name if there is one.

On the broken file it printed:

<orders>
  <order id="1">
    <customer>
      Ana
    </customer>
    <items>
      <item sku="A">
        <qty>
          2
        </qty>
      </item>
      <item sku="B">
        <qty>
          1
        </qty>
        </items>      <-- MISMATCH (open: item)
    <total>

The shape of the mistake is visible at a glance: the </items> sits at the depth of an item's children, and the marker says the open element is item. The error is on the line before, </qty> closed and item was never closed, which the column number on a 280 character line would have taken a minute to find.

At scale

The same function ran on the 28.6 MB file with 1.2 million lines in 2.0 s, and it found two markers:

      </state>      <-- MISMATCH (open: status)
      </order>      <-- MISMATCH (open: status)

The first is the real error: </state> where status is open. The second is a consequence: after the mismatch, </order> arrives while status is still considered open. Believe the first marker only, fix it and run again. This is the same rule as with a parser's error list.

For a big file, print only a window around the first marker, not the whole indented output, which is as large as the input. Pass a limit or break at the first mismatch and show the preceding 20 lines.

What the tokenizer gets wrong

It is a diagnostic and not a parser, and it is wrong in known ways.

**> inside an attribute value.** A tag such as <a href="x>y"> ends early in the pattern. Legal XML allows > in attributes, and the regex stops at the first one.

**Comments, CDATA and processing instructions are handled, and a stray < in text is not.** Text with a raw less-than, which is itself an error, will be split in the wrong place and throw off the indentation after it.

Entities are not looked at. & is plain text to it, so an ampersand error is invisible. Use a parser's message for those.

Names with colons and namespaces are treated as literal strings, which is right for matching tags.

Despite that, for the commonest structural mistakes (an unclosed element, a wrong closing name, two elements swapped) it points to the problem directly. The pattern it implements is the same as in an editor's bracket matching, and for XML that is often enough.

Other ways to see the shape

If the file is mostly right, an editor with XML folding shows the tags and highlights unmatched ones. xmllint --stream --noout reports the first error without building a tree (0.3 s on the large file), and grep -n '<status>' file | head with a count of </status> shows whether the numbers of open and close tags agree. A one-line count is a crude test: grep -o '<item' file | wc -l against grep -o '</item>' file | wc -l.

When the file does parse, formatting is simple. xmllint --format, lxml's pretty_print and ET.indent all work (pretty printing XML in Python). The caution there is mixed content, where added whitespace changes the text (why whitespace in XML sometimes matters). To read the first parser error in the usual way see XML parsing error not well formed and how to find the spot, and for the other messages fixing XML parse errors fast. The XML formatter in the browser formats well-formed documents and reports the error position for others. The libxml2 behavior is described in the xmllint manual, and Python's re module documentation covers the pattern syntax the tokenizer uses.

When a real parser error is enough

The file is not minified. With one element per line, the parser's line number is all you need.

The error is not structural. An ampersand, a control character, an encoding error and an undefined entity are better diagnosed by the parser's message.

You can regenerate the file. If the producer is yours, fix the generator and regenerate, instead of reading its output.

The file is small. Under a few hundred characters you can see the tags yourself.