Parsing and Writing XML in Python With ElementTree and lxml

Python can read and write XML with two libraries that look alike and behave differently. One is in the standard library, the other is a fast C-backed package with a much larger toolbox. The choice, and the handful of habits that keep XML code from producing bad data, are what this guide is about.

It is the overview for the Python group in this cluster, with the measurements from the individual articles gathered in one place and an end-to-end example at the end. The data are an orders file of 200,000 records (28.6 MB), a catalog of 50,000 books (8.3 MB), and a small library document.

The tools and what each is for

**xml.etree.ElementTree** is the standard library's tree API. parse, fromstring, find, findall, iter, SubElement, tostring. A small path language (ElementPath), no parent pointers, no validation, no XSLT. Always available.

**lxml** wraps the C library libxml2 and has nearly the same API, plus full XPath 1.0, getparent(), DTD, XSD and RelaxNG validation, XSLT 1.0, a recover mode, HTML parsing and control over the parser. It has to be installed. Its wheels bundle their own libxml2.

**xml.sax** is the event-driven parser, the lowest level and the lowest memory.

**xml.dom.minidom** is a DOM implementation from the standard library. Heavy, verbose, and worth avoiding for new code; its toprettyxml also damages mixed content.

**pandas.read_xml and to_xml** convert between XML and frames for flat records.

**xmltodict** converts between XML and dictionaries, with conventions for attributes and text.

**defusedxml** wraps the standard parsers with protections against entity attacks.

**feedparser** reads RSS and Atom into one normalized structure.

Most of the work is done by the first two, with the rest as specialists.

Reading: three styles

The whole tree

root = ET.parse("lib.xml").getroot()
for book in root.iterfind(".//book"):
    ...

The right style for files that fit comfortably in memory. It lets you ask many questions, navigate freely and use XPath on the whole document (reading XML with ElementTree). The habits that matter: handle None (a missing element and an empty one differ), test elements with is not None (an element with no children is falsy), convert strings to types in one place, and read text with itertext() when markup can sit inside prose.

The memory cost is high. For the 28.6 MB orders file the tree peaked at 344 MB in ElementTree and 430 MB in lxml, 12 to 15 times the file.

Streaming records

for event, el in etree.iterparse("orders.xml", tag="order"):
    ...process el...
    el.clear()
    while el.getprevious() is not None:
        del el.getparent()[0]

iterparse gives each element as it completes. With cleanup, peak memory was 22 to 43 MB for the same file, at the same speed (0.88 to 1.00 s against 0.79 to 1.09 s for the tree). Without the cleanup it was 345 MB: iterparse alone does not save memory. The two lines of cleanup are the whole trick (parsing large XML files with iterparse has the table for ElementTree, lxml and SAX).

SAX events

A handler with startElement, characters and endElement used 29 MB and 1.23 s. It is for cases where even iterparse is too coarse, or where you want one pass with exact control. For record-shaped files iterparse is easier to write and read.

ElementTree or lxml

Measured on the 200,000-order file:

operation                              ElementTree      lxml
parse                                     752 ms        262 ms
serialize the parsed tree               1,234 ms        113 ms
build 200,000 orders                      431 ms        424 ms
serialize the built tree                  650 ms         57 ms
Python loop over children, sum a field     32 ms        257 ms
xpath sum(//total)                           -           76 ms

lxml is faster at everything done in C: parsing, serializing, XPath, validation, XSLT. It is slower when Python code touches many nodes one at a time, since each access creates a proxy object. The right way to use it is to push work into XPath and XSLT and keep the Python loop short. ElementTree wins on availability and on small tasks. The features that decide the question are on lxml's side: XPath, parent pointers, validation, XSLT, error detail, HTML parsing (using lxml instead of ElementTree and why). A reasonable default is ElementTree until you hit a wall, then lxml; the move is mostly an import change.

Writing XML

The rule for writing is to use a serializer for anything built from data. The failures of string concatenation are not exotic: a raw ampersand, a control character, an invalid tag name, an attribute with a quote, or the wrong encoding. A serializer handles the first and the fourth. The others, it turns out, depend on the library:

• ElementTree accepted ET.Element("has space") and wrote <has space />, which no parser reads. It accepted an integer attribute and raised only at serialization (TypeError: cannot serialize 5). It wrote a raw vertical tab.

• lxml raised ValueError: Invalid tag name 'has space' at creation, refused the control character at assignment, and its E builder raised TypeError for a float argument.

So with ElementTree, validate names and convert values yourself, and always parse your output once in a test. With lxml, errors appear on the line that causes them (building an XML document from Python data). String templates with escape() took 85 ms for 200,000 orders against 430 ms for building a tree, and are reasonable for fixed, controlled documents.

On output: ask for UTF-8 (encoding="utf-8", xml_declaration=True) because ElementTree's default writes ASCII with character references. Format with ET.indent or lxml's pretty_print, which leave mixed content alone, and avoid minidom.toprettyxml, which split a paragraph into three lines and changed its text (pretty printing XML in Python). Indentation made the 28.6 MB file 22 percent bigger and gzip erased the difference.

Namespaces

A namespaced name is {uri}local in both libraries. Searches need a prefix map or the full name, attributes follow different rules from elements, and ElementTree invents ns0 prefixes on output until you call register_namespace, which is global. lxml keeps the document's prefixes. Stripping namespaces to make searches simple merges names from different vocabularies (handling XML namespaces in ElementTree). With lxml XPath, you cannot use a None prefix for the default namespace and you give it a name of your own (selecting elements in a namespace with XPath).

Validation and transformation with lxml

Two things that need lxml. Validation checks structure and types against a schema. On the 50,000-book, 8.3 MB catalog:

parse                                 193 ms
validate against XSD                   48 ms     (valid)
validate against DTD                   63 ms     (valid)
one price changed to GBP: XSD says    "attribute 'currency': [facet 'enumeration'] The value 'GBP'
                                       is not an element of the set {'USD', 'EUR'}", line 465

Validation is cheap relative to parsing, and the error carries the line. It is the best control on data from outside: parse, then validate, then use (well formed versus valid XML).

XSLT transforms a document with a stylesheet. The same catalog to 50,000 CSV rows took 54 ms with etree.XSLT, against 140 ms for a Python loop that builds tuples, and the stylesheet documents the mapping in a language made for it. XSLT 1.0 is dated and complete enough for flattening, filtering and reshaping.

From XML to tables

Three routes, in increasing order of control:

pandas.read_xml is one call for flat records and it took 2.2 to 3.4 s for 200,000 orders. It converts 001 to the integer 1 unless you set dtype, lets an element beat an attribute of the same name, keeps only the last of repeated children and drops nested structures without a warning (converting XML to a pandas DataFrame).

XPath per column is quick (1.13 s) and misaligns when a child is missing.

A loop per record, with findtext and explicit conversions, took 0.94 s, handles missing children and lets you build several tables for repeated elements (extracting values from XML into a table with XPath). This is the one to trust for data that matters.

For a specific kind of feed there are specific tools: feedparser for RSS and Atom, which recovered all four entries of a feed that ElementTree and lxml refused because of HTML entities, and flagged it with bozo (parsing an RSS feed with Python).

Safety

Parse untrusted XML as untrusted input. The baseline: cap the size of the body before reading it, parse with entity resolution off, DTD loading off and no network, do not call xinclude() on it, and treat the content as untrusted when you use it.

defusedxml rejected internal, external and parameter entities with EntitiesForbidden and, with forbid_dtd=True, any DOCTYPE, at a parse cost of 1,096 ms against 699 ms. A hardened lxml parser (resolve_entities=False, no_network=True, load_dtd=False, dtd_validation=False) left entities unexpanded at no measurable cost: 272 ms against 279 ms. Neither limits size, and ElementTree accepted a document nested 300,000 levels, so the cap and a depth check belong in front (safe XML parsing in Python with defusedxml).

An end-to-end script

Putting the pieces together, a feed loader for a catalog has this shape.

def load_catalog(path):
    data = open(path, "rb").read()                       # bytes, so the declaration is honored
    if len(data) > MAX_BYTES:
        raise ValueError("file too large")
    parser = etree.XMLParser(resolve_entities=False, no_network=True, load_dtd=False)
    root = etree.fromstring(data, parser)                # well-formedness
    if not XSD.validate(root):                           # structure and types
        err = XSD.error_log[0]
        raise ValueError(f"line {err.line}: {err.message}")
    rows = [(b.get("id"), b.findtext("title"), float(b.findtext("price")),
             b.find("price").get("currency")) for b in root]
    assert len(rows) == int(root.xpath("count(//book)")), "row count mismatch"
    return rows

Each line is a decision from an earlier section: read bytes, cap the size, a hardened parser, validate before use, convert types in one place, and compare the row count with an independent count. When the file is too large for memory, the loop becomes iterparse with clearing and the validation becomes a pre-pass with xmllint --schema --stream, or per-record validation. Wrap it with logging that includes the file name, the row count and the time, and quarantine failures with the error (fixing XML parse errors fast covers the triage routine for files that do not parse).

Testing XML code

Keep a folder of sample files next to the tests: a good one, one for each branch of your code (missing optional element, repeated element, empty element, namespaced), and the bad ones that must fail (truncated, wrong encoding, hostile). Assert on the results you extract, and add a round-trip test for writers: build, serialize, parse, compare. For writers, also validate the output against the schema in the test. For comparing documents regardless of formatting, use canonical XML (method="c14n" in lxml) after removing blank text, and remember that comments and processing instructions count unless you strip them.

Encodings and the declaration

Most encoding trouble in XML code comes from handing a parser text when it wants bytes. The declaration at the top of a file (<?xml version="1.0" encoding="ISO-8859-1"?>) tells the parser how to decode the bytes that follow. If you read the file yourself and decode it with the wrong codec, or decode it with the right one and then pass the resulting string to lxml, the parser either ignores the declaration or refuses the string outright. lxml raises ValueError: Unicode strings with encoding declaration are not supported for a str that carries a declaration, and the fix is one word: read in binary mode and pass bytes. ElementTree is more forgiving about strings, and that forgiveness hides the same mistake until a file with a non-UTF-8 declaration shows up (XML encoding declarations and the wrong encoding problem goes through the cases, and encoding mismatch errors in XML documents covers the failures).

The rule is short. Parse from bytes. Let the parser read the declaration. When you write, choose UTF-8 explicitly and write the declaration, so the next reader does not guess. When a file arrives with no declaration and no byte order mark, XML says it is UTF-8, and a file that is really Windows-1252 will fail on the first accented byte with a message about an invalid byte. That error is useful: it names the position, and charset-normalizer or chardet can suggest the real encoding, as long as you treat the suggestion as a guess and check it against a few known words.

A byte order mark in front of the declaration is legal for UTF-8 and breaks naive code that checks data.startswith(b"<?xml"). Parsers accept it. Hand-written sniffing should strip it first (XML prolog errors and the byte order mark).

A worked case: orders from a partner

Here is how the pieces fit on a realistic job. A partner sends a nightly orders file of about 30 MB, and you must load it into a table, reject anything malformed, and never let the load run out of memory. The synthetic file in this guide has the same shape: an order element with an id attribute, a customer, a status, a total and an optional note.

Step one is the contract. Ask for an XSD, or write one from three days of real files, and keep it in the repository next to the loader. The schema is the only description of the format that a machine can enforce, and the measured cost is small: 48 ms to validate 50,000 books.

Step two is a size and parse check on the raw bytes. Refuse files over a limit you chose on purpose. Parse with the hardened settings above. A file that is not well-formed goes to a quarantine folder with the parser message, the line and the file name, and the job continues with the next file. Nobody should have to reread a log to learn that a file was skipped, so the loader also emits a metric for rejected files.

Step three is validation. A well-formed file that fails the schema is a different kind of problem from a truncated one: the sender changed something, and the message with the line number is what you forward to them. Keep the first error and the count of errors, because a thousand errors with one cause are one problem.

Step four is the load. For a file that fits in memory, a per-record loop with findtext and explicit conversions is the clearest. For a file that does not, iterparse on the order tag with the two lines of cleanup keeps memory at tens of megabytes, and rows go to the database in batches inside a transaction, so a failure in the middle leaves nothing half-loaded. Compare the number of rows written with the number of order elements counted in the same pass. A mismatch is a bug, and the loader should fail loudly rather than report success.

Step five is the paper trail. Store the checksum of the raw file, the row count and the load time. When the partner disputes a number three months later, the checksum proves which file you loaded, and the original sits in cheap storage. Do not trust formatting for this purpose: a file reformatted by a tool has a different checksum and the same content, which is where canonical XML helps.

None of those steps needs more than the standard tools and one or two extras. The point of the example is the order: bytes, then limits, then well-formedness, then schema, then types, then load, then reconciliation. Each step catches a class of failure the previous one cannot see, and the loader is only as strong as the weakest link.

Mistakes that keep recurring

A short list, drawn from the tests in this cluster, of the errors that waste the most time.

Testing an element for truth. if book.find("title"): is false for an element with no children, even when it has text. The check you want is is not None, and for text, findtext with a default. This single habit explains many reports of "missing" data that was present (reading XML with ElementTree in Python).

Forgetting the namespace. A search for book finds nothing in a document where the elements are in a namespace, and the query returns an empty list rather than an error. When a result is unexpectedly empty, print the root tag first. If it starts with a curly brace, the namespace is the answer (handling XML namespaces in ElementTree).

Assuming the first match is the only match. find returns the first element. A record with two authors yields one, and pandas.read_xml keeps the last of repeated children. Nothing warns you. Count with findall when the schema allows more than one.

Trusting inferred types. A column of 001, 002 becomes the integers 1 and 2 in read_xml, and an empty element turns an integer column into floats. Convert in one place, on purpose.

Building a tree for a streaming job. A 28.6 MB file needs 344 to 430 MB as a tree. iterparse without cleanup uses just as much. The cleanup is what saves the memory.

Formatting the document and changing its meaning. Pretty printers add whitespace. Where the whitespace is part of the content (mixed content such as a paragraph with inline tags), indenting changes the text. Compare and hash with canonical XML after removing blank text, not with the formatted bytes. In my pipeline test the canonical hash of the original and of the re-parsed pretty version differed, which is the reminder that formatting is a change, even when it looks harmless.

Parsing untrusted input with the defaults. The defaults of modern parsers are much better than they were, and they still differ from library to library. Set the options explicitly and keep hostile fixtures in the tests.

Swallowing parse errors. A bare except Exception that returns an empty list turns a broken feed into "no new orders today". Catch the parser's error type, log the line and the file, and let the job fail or quarantine.

Performance notes

Some guidance that follows from the measurements.

Parsing is rarely the bottleneck in an ETL job. In the order test, lxml parsed 28.6 MB in 262 ms, and the work after parsing, such as database inserts, took far longer. Measure the whole job before choosing a faster parser.

Moving work into C is the main lever with lxml. A single XPath expression that sums a field took 76 ms, while the Python loop touching each node took 257 ms. The same idea applies to XSLT, which turned 50,000 books into CSV in 54 ms. When a loop over nodes is slow, look for the question that XPath or XSLT can answer in one call.

ElementTree is faster than lxml at a Python loop over children (32 ms against 257 ms in this test), because it stores children as plain Python objects, while lxml creates a proxy for each node you touch. The surprise is only that the faster library is slower for this particular access pattern. If your code is a tight loop over children and nothing else, the standard library may be the better choice even on a large file.

Serialization differs the most between the two: 1,234 ms against 113 ms for the 200,000 orders. A service that writes large documents benefits from lxml for that reason alone.

Compression changes the picture for transfer, not for parsing. Indentation made the file 22 percent bigger and gzip erased the difference, so format for readers and compress for the wire. The parse cost of a larger file is real, but whitespace between elements is cheap for the parser to skip.

Last, memory. If a job runs in a container with a memory limit, the tree sizes above decide it: a 30 MB file needs a gigabyte of headroom with some margin, or a streaming parse. Decide on that before the first deployment, not after the first kill by the out-of-memory handler.

How the articles in this cluster fit together

The Python group follows a path from simple to specific. Start with reading XML with ElementTree in Python for the tree API and its habits, then parsing large XML files with iterparse when the file stops fitting in memory. Using lxml instead of ElementTree and why lists what you gain by switching. The writing side is building an XML document from Python data, pretty printing XML in Python and handling XML namespaces in ElementTree. For tables there is converting XML to a pandas DataFrame, for feeds parsing an RSS feed with Python, and for hostile input safe XML parsing in Python with defusedxml. The query language that sits under all of it is covered in the XPath group, starting with XPath in Python with lxml.

When a file will not parse at all, go to the debugging group instead: fixing XML parse errors fast is the triage routine, and the individual error messages each have their own article. When the question is about the format itself, the basics group starts with what XML is good at and includes well formed versus valid XML and how XML namespaces work.

A checklist for XML code in production

Before an XML loader or writer goes live, go through this list.

1. Input is read as bytes, with a size limit chosen on purpose.

2. The parser is configured explicitly for untrusted data, and hostile fixtures are in the test suite.

3. Well-formedness errors are caught by type, logged with file and line, and the file is quarantined.

4. A schema exists and runs in the pipeline, with the first error forwarded to the sender.

5. Namespaces are handled with a prefix map or full names, never by stripping.

6. Types are converted in one place, identifiers stay strings, and missing values have a defined meaning.

7. Large files stream with cleanup, and a test proves memory stays flat.

8. Output is built with a serializer, names from data are validated, control characters are removed, and the result is parsed and validated in a test.

9. Row counts and checksums are recorded and compared.

10. Formatting is for readers; comparisons use canonical XML.

If every item has an answer, the code is in good shape. If one does not, that is the place to look first when something goes wrong, because the failures in this guide come from the gaps in that list.

A decision table

When a different tool is better than Python

The job is a one-off query. xmllint --xpath or xsltproc in the shell gets an answer without a script (running XPath from the command line with xmllint).

The data goes into a database. SQL engines with XML support can shred and query XML directly, and the load is often simpler there (storing and querying XML inside a database).

The structure is a document. Prose with inline markup is a job for XSLT or a document toolchain; flattening it to rows loses what it means.

Performance dominates. At billions of records, a compiled parser or a columnar pipeline beats any Python loop.

The reference documentation is the Python ElementTree module and the lxml documentation. To look at an XML sample in the browser, use the XML formatter, and the cluster starts at what XML is good at.