Benchmarking ElementTree lxml and iterparse on 127 MB

Which XML parser is fastest? The question comes up in every code review of a loader, and answers on the internet are decades old and for other versions of everything. This post measures four ways of doing one job on one file, on current versions. The result is less dramatic than the folklore: speed is close. Memory is where they differ, and a detail of the clearing pattern decides whether streaming actually saves any.

The job and the file

The file has 400,000 orders (127 MB), each with one to five order lines, 1,199,909 lines in total. The job is a typical extraction: for every line, read the qty attribute and the price child, and sum their product. It exercises attribute access, child lookup, numeric conversion and iteration over many small nodes. The correct total is 3,001,978,835.00, and every variant had to produce it.

The script takes a mode and a path and prints time and peak memory. The four modes:

import sys, time, resource
import xml.etree.ElementTree as ET
from lxml import etree

mode, path = sys.argv[1], sys.argv[2]
t0 = time.perf_counter()
total = 0.0; n = 0
if mode == "et-dom":
    root = ET.parse(path).getroot()
    for line in root.iter("line"):
        total += float(line.findtext("price")) * int(line.get("qty")); n += 1
elif mode == "lxml-dom":
    root = etree.parse(path).getroot()
    for line in root.iter("line"):
        total += float(line.findtext("price")) * int(line.get("qty")); n += 1
elif mode == "et-iter":
    for ev, el in ET.iterparse(path, events=("end",)):
        if el.tag == "line":
            total += float(el.findtext("price")) * int(el.get("qty")); n += 1
        if el.tag == "order":
            el.clear()
elif mode == "lxml-iter":
    for ev, el in etree.iterparse(path, events=("end",), tag="order"):
        for line in el.iter("line"):
            total += float(line.findtext("price")) * int(line.get("qty")); n += 1
        el.clear()
        while el.getprevious() is not None:
            del el.getparent()[0]
dt = time.perf_counter() - t0
rss = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024
print(f"{mode:12s} lines={n} total={total:.2f} time={dt:.1f}s peak_rss={rss:.0f}MB")

Results

et-dom       lines=1199909 total=3001978835.00 time=3.2s peak_rss=1389MB
lxml-dom     lines=1199909 total=3001978835.00 time=4.1s peak_rss=1922MB
et-iter      lines=1199909 total=3001978835.00 time=3.6s peak_rss=54MB
lxml-iter    lines=1199909 total=3001978835.00 time=4.3s peak_rss=22MB

Three observations, in order of how much they should change your code.

Memory differs by two orders of magnitude and time by a third. The slowest and fastest variants are within 35 percent of each other. The most and least memory-hungry differ by a factor of about 90. For anything beyond a few tens of megabytes, the choice is a memory decision (how much memory a DOM parser needs explains the multiplier).

The standard library was not slower. ElementTree finished in 3.2 seconds against lxml's 4.1 for the whole-tree case. Here the work is Python-level attribute access on a dozen nodes per order, and lxml pays for creating a Python proxy for each node it touches. lxml's advantage is in XPath, validation and tree manipulation, where its C implementation runs the loop. A script that only walks and sums is not where it shines.

The streaming variants are only as good as their clearing. et-iter peaked at 54 MB, not the 22 MB of lxml-iter. The reason is in how the loops release memory. ElementTree's clear() empties an order element, but the empty element object stays in its parent's children list, so the root accumulates 400,000 empty children, each a few dozen bytes. lxml has the same issue, and the while el.getprevious() is not None: del el.getparent()[0] loop deletes the earlier siblings so the root stays short. I removed that line and reran: lxml-iter peaked at 68 MB instead of 22 MB, in 5.3 seconds, because 400,000 empty order elements stayed attached to the root.

The clearing pattern

For lxml, use this skeleton. It handles records one at a time and keeps the tree around the current record small:

for _, el in etree.iterparse(path, events=("end",), tag="order"):
    handle(el)
    el.clear()
    while el.getprevious() is not None:
        del el.getparent()[0]

Three details are easy to get wrong. The tag="order" argument makes lxml deliver only the events you want, which is faster than filtering in Python. The end event is the right one: at start the children have not been read yet. And handle(el) must finish with the element before the clear(), because clearing erases the text and attributes you were about to read.

With ElementTree, the equivalent needs the root:

context = ET.iterparse(path, events=("start", "end"))
_, root = next(context)
for event, el in context:
    if event == "end" and el.tag == "order":
        handle(el)
        root.clear()

Calling root.clear() after each record drops all finished children at once. I ran this variant on the same file: it peaked at 22 MB, the same as lxml, in 4.1 seconds. The version in the benchmark clears each order instead, which is simpler and cost 32 MB more at this size. Prefer the root variant for files in the gigabytes.

A decision from the numbers

Put the three findings together and the choice is short. If the largest file you will ever see, times fifteen, fits comfortably in memory, use whichever API makes the code clearest. Most scripts are in this group, and for them the standard library is a sound default, with lxml added when you need XPath, schema validation or better error messages. If the file might not fit, stream with the clearing pattern, and test the memory with your largest sample before trusting it.

Do not pick a parser to save a second. The spread between the four variants was under one second on 127 MB, and a single slow database call in the loop would erase it. Pick to save memory, to simplify the code, or to get a feature, and keep the checksum in the test so any refactor that drops a record is caught on the next run.

Fair comparisons and what this does not show

A benchmark is a claim about one job. Change the job and the numbers move.

• A different task. If the loop did one XPath query per record, lxml would win on time. If it needed the parent of a node, iterparse would struggle.

• A cold file cache. These runs read from the page cache. A first read from disk adds seconds that no parser can remove.

• Other hardware. Absolute times will differ. Ratios are more portable than seconds.

• Different data shape. The file has 11 nodes per 330 bytes. A file of long text would show smaller memory gaps.

Run your own file through this script. Replace the loop body with your real extraction, keep the checksum, and compare. The measurement is ten minutes of work and replaces a debate.

Related reading

This is part of parsing big XML files fast. The streaming API is introduced in parsing large XML files with iterparse, and the library choice in using lxml instead of ElementTree. The loader that uses the pattern is in streaming XML into PostgreSQL without a DOM. The official references are the ElementTree iterparse documentation and lxml's incremental parsing guide. Check a sample element with the XPath tester.

When this benchmark will mislead you

You compare against a different language. A Java StAX or a Go decoder will have its own numbers. Compare within your stack.

Your records are nested deeply. The tag= filter and the sibling deletion assume records are children of the root. For deeper records, clear the correct ancestor.

Your task needs random access. Looking up a customer from an order, or an order from a line, needs the tree. Streaming is for single-pass work.

The file is not the bottleneck. If the work per record is a database round trip, parsing speed is a rounding error (loading large XML files into a database shows where the time goes).