Encoding Mismatch Errors in XML Documents From Mixed Sources

The previous article on declaration problems (XML encoding declarations and the wrong encoding problem) covers one file with one wrong claim. This one is about the harder cases: a file or a feed where the encoding is unknown, mixed, or was damaged earlier in a pipeline, and the question "can I detect it automatically?"

The short answer, from testing, is: sometimes, for some encodings, and the failures are the cases you most need help with. UTF-8 is easy to recognize and Latin-1 is not.

Why encoding errors come in kinds

When bytes meet the wrong decoder, one of three things happens, and it matters which:

1. The decoder rejects them (UTF-8 given Latin-1 bytes). You get an error at the first bad byte. Easy to notice.

2. The decoder accepts them and produces the wrong characters (Latin-1 given UTF-8 bytes, or cp1252 given Latin-1). No error. Hard to notice.

3. The bytes were already damaged (text decoded wrongly earlier, then encoded again as UTF-8). Every decoder accepts them, and the text is wrong in a systematic way.

An XML parser can only help with the first: it fails loudly if the bytes do not fit the declared encoding. The other two pass through and are discovered by a person reading the output. Detection tools exist for the second, with limits.

How well detection works

I generated short texts, encoded each in several encodings, and asked two detection libraries what they were:

sample  encoded as  charset_normalizer  chardet
fr      utf-8       utf_8               utf-8 (0.99)
fr      latin-1     cp1250              MacCyrillic (0.17)
fr      cp1252      cp1250              MacCyrillic (0.17)
de      latin-1     cp1250              Windows-1252 (0.07)
es      latin-1     cp1250              ISO-8859-1 (0.09)
ja      shift_jis   cp932               CP932 (0.70)
fr      utf-16      utf_16              UTF-16 (1.00)
ascii   any         ascii               ascii (1.00)

The pattern is clear. UTF-8 and UTF-16 were identified correctly every time. Valid UTF-8 for non-ASCII text is very unlikely to occur by chance, so detection is reliable. Japanese Shift JIS was identified as its superset CP932, which is fine. Pure ASCII is ASCII. And the single-byte Western encodings (Latin-1, cp1252) were not identified at all: charset_normalizer said cp1250 (Central European), and chardet said MacCyrillic with a confidence of 0.17, Windows-1252 at 0.07, or ISO-8859-1 at 0.09. Those confidence numbers are the tool admitting it is guessing.

The reason is structural. In a single-byte encoding every byte value is a valid character, so any bytes decode without error, and a statistical guess among the dozens of single-byte code pages is based on letter frequencies in a few dozen characters. Two of the Western ones also differ only in a few code points.

Short input makes it worse. The four-character word café in Latin-1 was reported as utf_16_be by charset_normalizer, and cp1252 text with curly quotes as cp1125. A tool that returns an answer for everything will return nonsense for small samples. And on the XML files from the declaration article:

utf8_decl_utf8         utf_8      correct
utf8_decl_latin1       utf_8      correct (the declaration was wrong)
utf16_bom_nodecl       utf_16     correct
latin1_decl_latin1     latin_1    correct
latin1_decl_utf8       cp1250     (true: latin-1)
cp1252_curly           cp775      (true: cp1252)
cp1252_decl_utf8       cp852      (true: cp1252)

Detection recovered the truth for every UTF-8 file, including the one with a lying declaration, which is useful. For the three single-byte files it produced three different wrong answers.

The approach that works: UTF-8 first

Since UTF-8 can be tested reliably, test it first. Decode strictly; if it succeeds, the file is UTF-8 (or ASCII). Only if it fails, fall back to the single-byte encoding your sources actually use:

def decode_xml_bytes(b):
    try:
        return b.decode("utf-8"), "utf-8"
    except UnicodeDecodeError:
        return b.decode("cp1252", errors="replace"), "cp1252"

Why cp1252 and not Latin-1 for the fallback? They agree for letters, and cp1252 also assigns the byte range 0x80 to 0x9F to curly quotes, the euro sign and the dash, where Latin-1 has control characters. Western text from Windows systems is cp1252 far more often than Latin-1, and decoding cp1252 as Latin-1 gives \x93 where a quote should be (the silent failure from the earlier article). Python's cp1252 decoder leaves five byte values undefined (0x81, 0x8D, 0x8F, 0x90, 0x9D), which is what errors="replace" is for.

The fallback should be chosen from knowledge, not detection. If you know the source is a Japanese system, it is Shift JIS or CP932; if it is a Vietnamese one, check for cp1258 or UTF-8. Keep the choice in configuration per source.

Files with more than one encoding

A feed assembled by concatenating pieces from different systems can contain several encodings. I made two lines, one in UTF-8 and one in Latin-1:

mixed = "<a>café</a>\n".encode("utf-8") + "<b>Zoë</b>\n".encode("latin-1")
mixed.decode("utf-8")
# UnicodeDecodeError: 'utf-8' codec can't decode byte 0xeb in position 18: invalid continuation byte

The error says which byte and where, which is how you find the second encoding. Three responses:

**errors="replace"** kept the file and produced <b>Zo�</b>: the letter is gone. Fine for a diagnostic, wrong for data you will use.

Per line, with a fallback. Decode each line as UTF-8 and use cp1252 for the lines that fail:

for line in mixed.splitlines():
    try: out.append(line.decode("utf-8"))
    except UnicodeDecodeError: out.append(line.decode("cp1252"))
# ['<a>café</a>', '<b>Zoë</b>']

Both lines recovered. This works if lines do not split characters and each line is one encoding. It does not work for a record that spans lines.

Fix the pipeline. A file with mixed encodings was assembled wrongly, and the only real fix is upstream: convert each source to UTF-8 on entry. Treat the per-line trick as a rescue for old data.

Note that the XML declaration cannot describe a mixed file. It names one encoding for the document, and the parser will apply it to all bytes.

Undoing double encoding

The third case is text that was decoded with the wrong encoding and then re-encoded, so that the damage is stored as valid UTF-8. café Zoë encoded as UTF-8 and decoded as Latin-1 becomes café Zoë, and if that is then written out as UTF-8 and read as UTF-8, it looks like café Zoë forever. The parser sees valid characters and no error. The fix is to reverse the mistake:

"café Zoë".encode("latin-1").decode("utf-8")      # 'café Zoë'

I applied it to a doubly damaged string (the mistake made twice, cafÃ\x83©...) and had to apply it twice to recover the text. Check the result for plausibility: if .encode("latin-1") fails (characters outside Latin-1), the text was not damaged in this way, and if .decode("utf-8") fails, it was damaged differently. The telltale for this problem is the pattern Ã or Â followed by an odd character, which appears whenever UTF-8 two-byte sequences for Western letters were read as Latin-1. A search for Ã. over a column of text finds it. The library ftfy automates the repair and I did not run it here.

Where double encoding comes from: a database connection set to Latin-1 reading UTF-8 columns, a CSV opened in a spreadsheet with the wrong encoding and saved, or an export script that decodes with the platform default.

Making a pipeline safe

Convert to UTF-8 at the edge, once, and record the source encoding and how you decided it. Validate after conversion: parse the XML, and scan the text for replacement characters (U+FFFD) and for the Ã pattern. Alert when either shows up. Never use errors="ignore", which drops bytes without a trace. Keep the original bytes next to the converted file, so a wrong decision can be reversed.

For the XML side of it see XML parsing error not well formed and how to find the spot and fixing XML parse errors fast. The CSV version of this problem is why a CSV shows strange characters instead of accents. The detectors are documented at charset-normalizer and chardet, and Python lists the codecs in its standard encodings table. To inspect a document that parses, use the XML formatter.

When detection is the wrong tool

You know the source. The vendor's documentation, or a sample opened by a person who reads the language, beats any statistical guess. Record it per source.

The file declares and agrees. If the declaration is present and the bytes decode strictly under it, there is nothing to detect.

The sample is short. A detector given a few words returns noise, as the four-character test showed. Feed it a large sample or skip it.

The text is damaged, not mis-declared. Detecting the encoding of double-encoded text says UTF-8 and is correct, and the text is still wrong. Look for the Ã pattern.