How Much Memory a DOM Parser Needs for a Large XML File

The first time a script that parses a 20 MB XML file ran out of memory on a server, the number surprised me. The file fits on a floppy disk's worth of modern storage many times over, and the process wanted a gigabyte. The multiplier is real, it is large, and it has a predictable cause. Knowing the number lets you decide, before you write the loader, whether a DOM parse is an option or a trap.

The test data

A generator writes orders with a customer, a date, and one to five order lines, each with a SKU, a quantity and a price. A typical order looks like this:

<order id="1" status="paid">
  <customer>customer-42</customer>
  <placed>2026-03-14</placed>
  <lines>
    <line sku="SKU1234" qty="3"><price>19.99</price></line>
  </lines>
</order>

Two files were made with the same generator and a fixed seed: 100,000 orders (32 MB, 300,251 lines) and 400,000 orders (127 MB, 1,199,909 lines). That is about 330 bytes per order, with many small elements: roughly eleven nodes per order including attributes and text. The task is deliberately small, to sum quantity times price over every line, so any difference comes from loading, not from the work.

The measurements

Each case ran in a fresh process:

The totals agreed in every case (3,001,978,835.00 for the large file), so the work was the same. The two things to notice are the ratios, which held steady as the file quadrupled, and the last row. Memory for a DOM parse grows linearly with the file, and the multiplier is large. A streaming parse uses a fixed amount no matter how big the file is, and took about as long.

I ran the same file through the 100,000-order case to check that the ratio was not an artifact of the small file. It was not: 11 and 15 times at 32 MB, and 10.9 and 15.1 times at 127 MB. If your documents look like these, the rule of thumb is ten to fifteen times for a DOM, depending on the library.

Where the multiplier comes from

The text in the file is a minority of what the parser stores. A DOM keeps a node object for every element, attribute and text run. Each object carries a tag name reference, pointers to parent, siblings and children, an attribute dictionary or list, and bookkeeping for the garbage collector. In CPython, even an empty object costs dozens of bytes, and a Python-level wrapper adds more.

Count the nodes in one order: an order with two attributes, customer and placed with a text node each, lines, and per line a line element with two attributes, a price element and its text. That is over a dozen nodes for 330 bytes of text. If each node costs 100 to 150 bytes once wrappers are included, the tree is several times the file. The measured 11x for ElementTree is consistent with that.

lxml costs more here for a specific reason. It keeps the C tree that libxml2 built, and it creates Python proxy objects when you touch an element. For code that walks everything, as this test does, both representations are alive at once. The compensation is speed on queries: lxml's XPath and iter() run in C, and for repeated lookups on one tree it is much faster than ElementTree. If memory is what you lack, the standard library is the lighter choice. If the document fits and you query it heavily, lxml pays for itself.

Two properties make the multiplier worse. Files with many tiny elements cost more per byte than files with long text, because the per-node cost is fixed. A 100 MB file of prose paragraphs may need three times its size, and a 100 MB file of one-attribute elements may need twenty. And parsing from a string rather than a file holds a second copy of the text, so fromstring(open(f).read()) adds the file size to the total. Pass the file name or a file object so the parser reads in blocks.

What to do with the number

Use the multiplier to make the decision early. Multiply the largest file you expect by fifteen and compare it with the memory the process may use.

• Under a quarter of the memory limit: a DOM is fine. Use it, because the code is shorter.

• Between a quarter and the limit: you are one bigger file away from a failure. Switch to streaming now, while the cost is low.

• Over the limit: streaming is not optional. Use iterparse or SAX (parsing large XML files with iterparse).

Containers make this sharper than it looks. A process limited to 512 MB by an orchestrator is killed with no Python traceback when it crosses the line, and the log says only that the container exited with code 137. A file that parsed on your laptop with 16 GB dies in production at 35 MB of input.

The streaming version of this test used 22 MB at its peak, and the cost was a more careful loop: clear each processed element and delete earlier siblings so the root does not accumulate children. The pattern is in benchmarking ElementTree, lxml and iterparse, and loading large XML files into a database applies it to a loader.

Related reading

This post belongs to parsing big XML files fast. The memory of a limited server shapes the choice in streaming XML into PostgreSQL without a DOM. Python's own notes on the two APIs are in the ElementTree documentation, and lxml's incremental parsing is described on its parsing page. To look at the structure of a sample before you size it, use the XML formatter.

When the memory multiplier does not apply

Other languages and parsers. The ratios here are for Python objects. A DOM in Java, .NET or C++ has different per-node costs, usually lower than CPython's but the same order of magnitude. Measure your own stack on your own file. SAX and StAX parsers hold almost nothing, in every language.

Native binary trees. A parser that keeps a compact tree in C structures without per-node language objects, such as libxml2 used directly, needs about two to five times the file size.

Small files. Under a few megabytes, the multiplier is irrelevant. Nobody tunes a configuration file.

Files with large text blocks. A document that is mostly paragraphs has few nodes, so its multiplier is close to the text size plus a small overhead. The measurements above are for record-style data.