Invalid Character in XML and Stripping Control Characters

A file that parsed fine last week fails with:

lxml:         PCDATA invalid Char value 11, line 420004, column 19
ElementTree:  not well-formed (invalid token): line 420004, column 18
xmllint:      parser error : PCDATA invalid Char value 11

The text contains a character that XML 1.0 does not allow. The number in the lxml message is the code point: 11 is a vertical tab, which looks like nothing on screen, so you stare at a line that seems fine. This is the invisible-character error, and it is one of the three most common reasons a good feed breaks overnight.

I put a vertical tab into one of 200,000 orders in a 28.6 MB file and tried the fixes.

What XML 1.0 allows

The set of legal characters is defined in the specification as a few ranges:

#x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF]

In words: tab, newline, carriage return, and everything from space upward except the surrogate range and the two noncharacters U+FFFE and U+FFFF. The characters U+0000 to U+001F other than the three whitespace ones are out. That covers NUL, backspace, vertical tab, form feed, escape and the rest.

I tested individual characters in the text of an element:

raw U+0000   error      raw U+0008   error      raw U+000B   error
raw U+000C   error      raw U+001F   error      raw U+007F   accepted
raw U+0085   accepted   U+FFFE       error      U+FFFF       error

U+007F (DEL) and U+0085 (a next-line control in the C1 range) are allowed by XML 1.0 even though they are controls, which is a quirk of the definition. Everything below U+0020 that is not tab, newline or carriage return is not.

Escaping does not help

The reflex is to escape the character, and it fails for these:

<a>&#11;</a>     ParseError: reference to invalid character number: line 1, column 3
<a>&#xB;</a>     the same
<a>&#x9;</a>     ok (a tab is legal)

A character reference to a forbidden character is just as forbidden. In lxml the message for it is xmlParseCharRef: invalid xmlChar value 11. XML 1.0 has no way to represent a vertical tab in text. (XML 1.1 allows references to most control characters, and almost nothing implements it.) If the data really contains a control character that matters, the format has to carry it another way: base64, a hex string, or a replacement such as \u000b text.

Find them

Before you fix, count the problem. A scan for forbidden bytes over the 28.6 MB took 0.09 s:

BAD = re.compile(rb"[\x00-\x08\x0b\x0c\x0e-\x1f]")
len(BAD.findall(data))     # 1

One occurrence. That is the typical shape: a few stray characters among millions of good ones. Print their positions to see what they belong to:

for m in BAD.finditer(data):
    line = data.count(b"\n", 0, m.start()) + 1
    print(line, hex(data[m.start()]), data[m.start()-20:m.start()+20])

The context usually answers where they came from: a name with a vertical tab in the middle of a word is a paste from a PDF, and a field full of \x00 is a binary value stored in a text column. This works on bytes for UTF-8 and ASCII-compatible data, where the control characters are single bytes below 0x20 and cannot be part of a multibyte sequence. For UTF-16 or other encodings, decode to a string first.

Remove them

The fast way is to delete the bytes:

fixed = data.translate(None, bytes(range(0, 9)) + b"\x0b\x0c" + bytes(range(14, 32)))

bytes.translate(None, delete) removed them in 0.032 s for 28.6 MB. The regular expression version, BAD.sub(b"", data), took 0.15 s and gave identical output. After the fix the file parsed in 0.62 s. For a string, a character class that keeps only the legal characters is the safe form, because it also removes lone surrogates and noncharacters:

ILLEGAL = re.compile("[^\u0009\u000A\u000D -퟿-�\U00010000-\U0010FFFF]")
ILLEGAL.sub("", "ok\x00\x08text\x0b")      # 'oktext'

Deleting is not the only choice. Replace the character with a space where it separated words, or with ? or U+FFFD to keep a visible trace. Think about what the field is. A vertical tab inside a name is noise and deleting it loses nothing. In a hash, a checksum or a binary field it is data, and deleting it destroys the value. Decide per field, and keep the original bytes in the archive.

Log what you did: how many characters, which code points, on which lines, which file. A feed that suddenly has 10,000 control characters when it normally has two is telling you something about the source.

Prevent it

The best place to fix this is where the text enters. If you generate XML, sanitize every string before it goes into the serializer. Python's lxml raises ValueError: All strings must be XML compatible: Unicode or ASCII, no NULL bytes or control characters if you try to set text containing them, and ElementTree writes them out happily, which produces a file that cannot be read back. A common pattern is a small clean(text) function, called once on every user-supplied or database-supplied string.

If you receive the data from someone else, validate on entry: parse the file at the boundary, reject or quarantine failures with the reason, and tell the sender. A pipeline that cleans silently teaches the sender nothing.

Where do the characters come from? Copy and paste from PDFs and word processors (form feeds and vertical tabs), legacy databases where a field held \x00 as a terminator, binary data in a column declared as text, serial or terminal logs with escape sequences, and exports from systems that treat XML as "text with angle brackets". Each of them is fixed at its source or at the first conversion.

Related checks

The generic not well-formed (invalid token) from ElementTree is why the other tools are worth running: xmllint --noout names the cause. If the same file also contains raw ampersands, fix one at a time, since the parser stops at the first (XML parsing error not well formed and how to find the spot, fixing XML with unescaped ampersands from a legacy export). The list of legal characters appears next to the other syntax rules in escaping special characters and in the cheat sheet, and the rest of the common failures are in fixing XML parse errors fast. The definition is in the XML 1.0 character range production, and Python documents bytes.translate in its built-in types reference. The XML formatter is a quick way to check a small cleaned snippet.

When removing the character is wrong

The character is data. A binary identifier or a checksum that includes control bytes must be encoded (base64, hex), not stripped.

The document is not XML 1.0. Some formats use 1.1, or use the control characters on purpose. Check the declaration and the standard.

The text is in a different encoding. Bytes that look like control characters may be the result of decoding UTF-16 or a double-byte encoding as if it were ASCII. If many of the offenders are \x00 between letters, the file is UTF-16.

The input is binary. If a "text" field is mostly control characters, it was never text. Fix the source field, not the parse.