Fixing XML With Unescaped Ampersands From a Legacy Export

A legacy system exports XML by concatenating strings. Customer names with an ampersand go in as typed: Smith & Sons, R&D, AT&T. The first time one reaches a real XML parser, the whole file is rejected:

not well-formed (invalid token): line 4, column 19      (ElementTree)
xmlParseEntityRef: no name, line 4, column 22           (lxml)

The cause is the character that starts every entity reference. A bare & in text is an error because the parser takes it as the beginning of &something;. The proper fix is a producer that escapes, and while you wait for one you repair the file.

I built a 29.2 MB export with 200,000 orders in which the ampersands were raw and repaired it.

The naive fix, and where it breaks

The obvious repair is to replace every ampersand:

fixed = text.replace("&", "&")

On a file in which every ampersand is raw, this is correct. The legacy export was like that, and the replacement worked. Many real files are not that clean. Some of the content passed through an escaping step and some did not, and a file contains both R&D (right) and Smith & Sons (wrong). In the mixed file:

already correct &      15,385
raw ampersands             22,755
after text.replace:        15,385 occurrences of &

The correct ones became &, which the parser reads as the literal text &. The file parses, and the data is now wrong: every R&D reads as R&D in your database. This is worse than a parse error, because nothing fails.

The safe fix: escape only the ampersands that are not entities

An ampersand is part of a valid reference if it starts one of the five predefined entities or a numeric character reference. Escape the others, using a negative lookahead:

PAT = re.compile(r"&(?!(?:amp|lt|gt|quot|apos|#[0-9]+|#x[0-9a-fA-F]+);)")
good = PAT.sub("&", text)

The pattern says: an ampersand that is not followed by one of the legal references and a semicolon. On the mixed file it escaped 22,755 raw ampersands and left the 15,385 existing entities alone, with no & afterwards. On the legacy file it escaped all 215,385 in 0.08 s (the file is 29.2 MB), and the repaired text parsed into 200,000 orders in 0.77 s. The customer text came out as Tom & Jerry 0 and the note as R&D, exactly as they were typed.

The repair is idempotent: running it twice gives the same result, because after the first run no raw ampersand is left, and the entities are protected by the lookahead. That makes it safe to put in a pipeline step that cannot know whether a given file was already fixed.

Cases the pattern has to settle

Short test strings showed what the regex does at the edges:

'AT&T'                 -> 'AT&T'
'Q&A & more'       -> 'Q&A & more'
'a < b'             -> 'a < b'
'© ©'        -> '© ©'
'R&D;'                 -> 'R&D;'
'x &y; z'              -> 'x &y; z'
'AT&T;'                -> 'AT&T;'
'&'                    -> '&'
'http://x?a=1&b=2'     -> 'http://x?a=1&b=2'
'&#x;'                 -> '&#x;'

Most of these are what you want. Raw ampersands before letters, spaces and equals signs become &, entities stay. A few deserve thought.

**©** was escaped to ©, which reads as the literal text ©. If the source meant the copyright sign, that is a loss, and the right repair is a conversion to the numeric reference ©. The HTML-aware version in the article on undefined entities does that (undefined entity errors in XML and the ampersand problem). Which behavior you want depends on whether the legacy system wrote HTML entities. If it did, convert known names first, then run the ampersand pass for the rest.

**R&D; and AT&T;** (a semicolon after a name-like piece) are not entities in XML unless they are one of the five, so the regex escapes them. In a legacy export that treats & as a plain character, that is the correct reading. If it came from an HTML source, &T; is an undefined entity, and treating it as text is still the only reasonable choice.

URLs with query strings (a=1&b=2) are the most common legitimate case, and they need & in XML attributes and text alike. The regex fixed it.

A semicolon far from the ampersand is not a problem: the lookahead requires the semicolon right after the entity name.

Already-escaped text inside CDATA is a case where the regex does harm. Inside <![CDATA[...]]>, an ampersand is literal and does not need escaping, and escaping it changes the content to show & literally. If the file uses CDATA sections, skip them. A simple way is to split the text on <![CDATA[...]]> and apply the pattern only to the parts outside. I did not test a file with CDATA, so check yours.

After the repair

Verify, in this order. Parse the file. Count entities: & should have increased by exactly the number of raw ampersands the pattern replaced. Search for the damage signature, &, which should be zero unless the source had it. Spot-check a handful of the changed values in context against the original. And keep the original: the repair is a transformation of someone else's data.

assert PAT.sub("&", good) == good                  # idempotent
assert "&" not in good or "&" in text  # no new double escapes
ET.fromstring(good.encode())                            # parses

For a very large file, process in chunks to avoid reading it all, with one precaution: an entity can be split across a chunk boundary (&am at the end of one chunk and p; at the start of the next). Cut chunks at a newline, or at the end of a record, not at a fixed byte count.

Fix it at the source

A repair step is a patch. A file that needs 215,385 repairs has been generated by code that does not escape, and the next file will need the same. Tell the producer and give them the three-line explanation. If they cannot change it, put the repair in a named step of your pipeline, log how many ampersands it fixed each run, and alert on a jump. A sudden rise means the content changed, and a drop to zero means the producer fixed it.

If you generate XML yourself, use a serializer that escapes (ElementTree, lxml), and never build documents by string concatenation (XML entities, CDATA and escaping special characters). For the broader list of parse errors, see fixing XML parse errors fast. The grammar of references is in the XML Reference production and Python documents lookahead assertions in the re module. To look at a small repaired fragment, use the XML formatter.

When automatic repair is the wrong answer

The data is financial or legal. Silent edits to a record are not acceptable. Reject the file, ask for a corrected one, and use repair only for investigation.

The file is not text with stray ampersands. If the damage is wider (broken tags, encodings), fixing ampersands first does not make the file valid, and the later errors hide behind it.

The ampersands carry meaning. In some formats & is a separator or an escape. Escaping it changes the data. Check what the producer intended.

You can ask for a better export. A corrected export costs the sender a few minutes and saves every consumer the same patch.