Undefined Entity Errors in XML and the Ampersand Problem
Fix undefined entity and Entity not defined errors in XML from HTML entities like nbsp and eacute: convert to numeric references, a DOCTYPE, or lxml recover.
A document that works in a browser fails in an XML parser with an error like these:
ElementTree: ParseError: undefined entity: line 1, column 6
lxml: XMLSyntaxError: Entity 'eacute' not defined, line 1, column 15
xmllint: parser error : Entity 'nbsp' not definedThe cause is almost always the same: the text came from HTML, or from a system that writes HTML-style entities, and XML does not know them.
An entity is a name for a piece of text, written between an ampersand and a semicolon. HTML defines over two thousand of them ( , ©, é, —, …). XML defines exactly five, and the rest must be declared in a DTD or replaced. A parser that meets é with no declaration stops. This is not a quirk of one library; it is the rule in the specification.
I wrote a test document that mixes legal XML entities, HTML ones and a nonsense name, and tried each repair on it, and on a 7.6 MB document of 200,000 paragraphs.
What is and is not defined
The five names that always work:
< > & " 'Numeric character references also always work, with no declaration: é and é are both é. Everything else needs a declaration.
So the most direct fix is to change the HTML names into numeric references before parsing. The Python standard library has the table. html.entities.name2codepoint has 252 names (the HTML 4 set, including all the ones above), and html.entities.html5 has 2,231 (the HTML5 set, keyed with the trailing semicolon, such as check;).
Repair one: convert named entities to numeric references
A function applied by re.sub handles each &name; it finds:
from html.entities import name2codepoint
XML5 = {"amp", "lt", "gt", "quot", "apos"}
ENT = re.compile(r"&([A-Za-z][A-Za-z0-9]*);")
def fix(m):
name = m.group(1)
if name in XML5:
return m.group(0) # leave the legal ones alone
if name in name2codepoint:
return f"&#{name2codepoint[name]};"
return "&" + name + ";" # unknown: make it literal text
fixed = ENT.sub(fix, doc)On the test text the result was:
<p>Café © 2026 — a b <ok> & &unknown; …</p>and it parsed to the accented text, with &unknown; kept as literal text. Three rules did the work. The five legal entities were left alone (touching & would double-escape it). Known HTML names became numeric references. And a name that is neither became a literal ampersand followed by the text, so that a stray &foo; in someone's prose survives as the characters &foo; and not as a parse error.
On the 7.6 MB document with 200,000 paragraphs, each containing é, and ©, the regex repair took 0.29 s and the parse of the repaired text 0.17 s. It is a two-pass approach, and fast enough that you can apply it to a feed on every run.
If the source is real HTML5 text with names outside the HTML 4 set (✓, ♥), use the larger table. Its keys carry the semicolon (html5["check;"] is ✓), and a name may map to more than one character, so build the numeric reference from each code point. A simpler route for text is to unescape and re-escape: html.unescape(s) turns every HTML entity into its character, and then you escape only &, < and > again. That works if the input is plain text. It cannot be used on a whole XML document because it would also unescape < inside the content, turning it into markup.
Repair two: recover mode (and why not)
lxml has a recovery mode that continues past errors:
etree.fromstring(doc.encode(), etree.XMLParser(recover=True)).text
# 'Caf 2026 ab ok 'It parsed and removed every undefined entity. é, ©, the dash, the non-breaking space and the ellipsis are all gone, and there is no error. The output is plausible, shorter, and wrong. Recovery is useful to look at a damaged file, and a loader that uses it loses characters without telling anyone. xmllint --recover does the same thing.
Repair three: declare the entities
A document can declare the entities it uses in its own DOCTYPE:
<!DOCTYPE p [<!ENTITY nbsp " "><!ENTITY eacute "é">]>
<p>Café x</p>ElementTree parsed it and returned 'Café\xa0x' with a real non-breaking space. This is how XHTML documents work, with a DOCTYPE that points to a DTD defining the HTML entities. It is a legitimate fix for files you write yourself. It is the wrong one for input from outside: it turns on the DTD machinery, which can bring in external entities and expansion attacks (what a DTD is and why you rarely need one). If the file is untrusted, parse it with a library that rejects DOCTYPEs and do the numeric conversion before it.
Preventing it where it starts
Entities show up in XML for a few reasons, and fixing the source is better than repairing every file.
Content copied from HTML or a CMS. A rich text editor stores and ’ in a field, and an exporter writes the field into XML without converting. The exporter should decode HTML entities and let the XML serializer escape what needs escaping.
A template engine that escapes for HTML. Templates for web pages escape with HTML entities. If the same engine builds XML, it produces entities XML does not know. Use an XML serializer or the engine's XML mode.
Double escaping. Text that contains &nbsp; was escaped twice, and it reads as the literal characters . The parser will accept it; the reader sees the wrong text. Look at the extracted value, not just whether the file parses.
Data from a database. Legacy systems sometimes stored text already HTML-escaped. Decode at the boundary, once.
When you generate XML, use a serializer, and never concatenate strings (XML entities, CDATA and escaping special characters covers what must be escaped and why).
Check the result
After a repair, check three things. The document parses. A count of characters of interest is right (the number of é in the output equals the number of é in the input). And nothing was double escaped: search the repaired text for &# and &amp;, which would show a repair that touched something it should not have. A good test is to run the repair twice: the second run should change nothing.
For raw ampersands without a semicolon (Smith & Sons), which look similar and have a different cause, see fixing XML with unescaped ampersands from a legacy export. Both are on the list in fixing XML parse errors fast. The Python tables are documented in the html.entities module, and the XML rules in the predefined entities section of the specification. The HTML entities tool looks up a name quickly.
When the entity is not the real problem
The document is HTML. If you are parsing a web page with an XML parser, use an HTML parser; lxml.html and html.parser accept entities, unclosed tags and the rest.
The entity is declared in a DTD you do not load. A document with a DOCTYPE that points at an external DTD uses its entities, and a parser that does not load external DTDs reports them as undefined. The fix is to get the entities, or to convert them, not to enable network access.
It is a raw ampersand. Entity 'x' not defined needs a name and a semicolon. xmlParseEntityRef: no name is the other error, for & followed by a space.
The character exists but is not allowed.  is a reference to a character XML forbids, and its error is different (reference to invalid character number).
