Parsing Big XML Files Fast Without Running Out of Memory

A large XML file fails in a predictable order. First the script is slow, then it uses all the memory, then it is killed by the operating system with no useful message, and finally someone suggests a bigger server. The bigger server postpones the failure until the next, larger file. This guide is the path out: what the cost of a big XML file really is, how to measure it on your own data, which techniques remove it and in what order to try them. Every number comes from one set of test files run on one machine, and the scripts are shown so you can repeat them with yours.

What "big" means for XML

Big is not a size. It is a ratio between the file and the resource that runs out first, and for XML that resource is almost always memory. A 50 MB file is nothing for a laptop and a catastrophe for a container limited to 512 MB, and the reason is a multiplier that most people do not know exists.

Three costs can dominate, and a first step is to find which one is yours.

Memory. A parser that builds the whole tree holds a node object for every element, attribute and piece of text. On the test file, that was 11 times the file size for ElementTree and 15 times for lxml. This is the cost that kills processes.

CPU. Parsing itself is fast. The 127 MB file parses in three to four seconds in Python, so CPU time is rarely a problem until the file is several gigabytes or the work per record is heavy. When people say XML is slow, they usually mean the work they do after parsing.

Round trips. If every record leads to a database call, an HTTP request or a file write, the time is in the waiting. Naive loaders spend 95 percent of their runtime on network latency, and parsing improvements change nothing.

The remedies differ for each, so measure first. The cheapest measurement is two numbers: peak memory and wall time of a bare parse that does nothing with the records. If memory is the problem, you stream. If the bare parse is fast and the full job is slow, the problem is in the work, and you look at batching and parallelism.

Measure before you tune

A bare parse on the test file shows what the format costs on its own. These four runs sum quantity times price over every order line, which is a stand-in for any extraction:

et-dom       lines=1199909 total=3001978835.00 time=3.2s peak_rss=1389MB
lxml-dom     lines=1199909 total=3001978835.00 time=4.1s peak_rss=1922MB
et-iter      lines=1199909 total=3001978835.00 time=3.6s peak_rss=54MB
lxml-iter    lines=1199909 total=3001978835.00 time=4.3s peak_rss=22MB

Time is within a third across all four. Memory is different by a factor of ninety. That gap is the whole story of big XML, and it is what the rest of the guide exploits. Benchmarking ElementTree, lxml and iterparse has the script and the details, and how much memory a DOM parser needs explains where the multiplier comes from and how to size a decision with it.

Notice the total at the end of each line. A checksum computed during the run is the single most useful habit for this kind of work. If a faster variant gives a different total, it dropped or doubled records, and no amount of speed makes up for that. Compute a count and a sum in the baseline once, keep them, and compare in every variant.

Step one: stop building the tree

For a file of records, the fix is to process each record as it passes and discard it. In Python this is iterparse:

from lxml import etree

for _, el in etree.iterparse(path, events=("end",), tag="order"):
    handle(el)
    el.clear()
    while el.getprevious() is not None:
        del el.getparent()[0]

The loop gets each order when its closing tag has been read, so the whole record is available; you handle it; then you release it. The while line matters. Without it, lxml keeps 400,000 empty order elements attached to the root, and the peak was 68 MB instead of 22 MB in my run. With ElementTree, the equivalent is to call root.clear() after each record, which also gave 22 MB. A plain clear() on each element left ElementTree at 54 MB.

Streaming gives you a constant memory ceiling, independent of file size. It does not give you random access: you cannot ask for the customer of an order whose id you saw a minute ago unless you stored it. If the job needs several passes, run several streams. If it needs to join data from different parts of the file, extract the keys while streaming and join in a database, where joins are a solved problem.

The same idea exists in every language: SAX and StAX in Java (streaming large XML files in Java), XmlReader in .NET, encoding/xml tokens in Go. If your language is Python, parsing large XML files with iterparse is the longer walkthrough.

Choosing a library once you stream

With streaming in place the library choice stops being about memory. ElementTree is in the standard library, was slightly faster on this plain loop (3.6 seconds against 4.3) and needs no installation. lxml adds XPath, XSLT, schema validation, better error messages and a tag= filter that skips events you do not want. If your records need an XPath query each, lxml's C implementation wins. If they need attrib and find, the standard library is as good. Pick lxml when you need its features (using lxml instead of ElementTree lists them) and the standard library otherwise.

For one-line questions, you may not need Python at all. These ran on the 127 MB file:

grep -c '<order '                    0.3s  -> 400000
xmllint --noout --stream             1.1s  (13 MB)
xmllint --noout (DOM)                3.5s  (1,908 MB)

grep counted the orders in 0.3 seconds, which is the fastest honest answer to "how many records are in this file" provided the file is machine-written and the string cannot occur anywhere else. xmllint --stream checked that the whole file is well-formed in 1.1 seconds and 13 MB, and without --stream the same check built the tree and needed almost 1.9 GB. The flag costs nothing. Use it for any file you do not need to query. Beware xmllint --xpath 'count(//line)' on big results: it printed 1.19991e+06, because XPath numbers are floating point and the tool formats them with six significant digits. xmlstarlet sel -t -v 'count(//line)' printed the exact 1199909. Wrap a count in string() or use another tool when the exact number matters. Counting nodes in XML from the shell compares the options.

Step two: load into a database properly

Most big XML files end up in a database, and this is where the time goes. A loader that parses the whole tree and runs one INSERT per row spends its time on 1.5 million network round trips and holds the tree in memory. The two fixes stack: stream the parse, and use the database's bulk path.

PostgreSQL's bulk path is COPY. With psycopg 3 it is a context manager, and rows go in as the parser yields them:

with cur.copy("COPY xo_orders (id,status,customer,placed) FROM STDIN") as co, \
     conn2.cursor().copy("COPY xo_lines (order_id,sku,qty,price) FROM STDIN") as cl:
    for _, el in etree.iterparse(path, events=("end",), tag="order"):
        oid = int(el.get("id"))
        co.write_row((oid, el.get("status"), el.findtext("customer"), el.findtext("placed")))
        for line in el.iter("line"):
            cl.write_row((oid, line.get("sku"), int(line.get("qty")), line.findtext("price")))
        el.clear()
        while el.getprevious() is not None:
            del el.getparent()[0]

Results on the 400,000-order file, with the database on the same Docker network:

orders=400000 lines=1199909 sum=3001978835.00 time=6.5s peak_rss=44MB
naive DOM + row INSERT (100,000 orders only): time=38.5s peak_rss=519MB

The streaming loader wrote 1.6 million rows in 6.5 seconds with 44 MB of memory. The naive loader took 38.5 seconds and 519 MB for a quarter of the data, which scales to about two and a half minutes and 2 GB for the full file. The sum computed in SQL after the load matched the sum computed in Python while benchmarking, which closes the loop on correctness. Streaming XML into PostgreSQL without a DOM has the full script and the trap that cost me an hour: two COPY commands cannot share one connection, and the second one hangs forever without an error.

For MySQL, SQLite and DuckDB the principle is the same with different calls: batch the inserts in a transaction, or write a CSV and use the bulk loader. Work out the row shape before you write the loader (flattening repeating XML elements into rows) and keep a rejects path for rows that fail a type check, because one bad value aborts an entire COPY.

If the XML must stay in the database as XML, the cost moves to indexing. On 50,000 small documents in an xml column, a lookup by one value took 245 ms as a scan and 1 ms with an expression index on the extracted value, and loading took 2.8 times as long with the index in place. A stored generated column with an ordinary B-tree index gave the same speeds with clearer queries. Indexing XML columns in PostgreSQL without wrecking inserts goes through the plans and the trade-off.

Step three: use all the cores, when it pays

A single XML parse runs on one core. If you have ten, nine are idle. A file that is a list of independent records can be cut into several valid files, and the parts processed by a pool of processes. On the test file, splitting into four parts took 3.4 seconds, and processing them took:

serial   : lines=1199909 total=3001978835.00 4.1s
pool(2)  : lines=1199909 total=3001978835.00 2.1s speedup 2.0x
pool(4)  : lines=1199909 total=3001978835.00 1.1s speedup 3.8x

The speedup is nearly linear, and the totals are identical. But add the split: 3.4 + 1.1 is longer than 4.1. For a job as light as this one, parallelism loses. It wins when the work per record is much heavier than parsing, when the same parts are processed repeatedly, or when the files arrive pre-split, as daily or regional deliveries often do. Parallel processing of split XML files has the splitter and the pool, and the rule that the workers must be processes, since threads share the interpreter lock.

Never split with a text search on a byte offset. A boundary found by searching for <order fails the day that string appears in a comment or a CDATA section, and it produces a corrupt record without an error. Split with a real parser, once, and verify that the record counts add up.

Step four: shrink the file

XML compresses well, because its names repeat. The 132 MB file became:

Pick gzip for universal support, zstd when you control both ends and want speed, xz for cold archives. You do not have to unpack to read: iterparse accepts any file object, so gzip.open(path, "rb") works, and it summed the whole file in 4.3 seconds against 4.2 for the plain one, at the same 22 MB. A shell pipe from zcat into a script that reads standard input finished in 2.5 seconds, because decompression and parsing ran on separate cores.

What compression does not do is reduce memory for a tree, and it does not allow random access to a single gzip stream. Compressing XML archives and reading them in place covers block formats, tar versus zip, and the decompression bomb, where a small archive expands to gigabytes. Always cap the size you will read from an untrusted source (limiting XML size and depth for uploads).

What a memory limit really looks like

On a laptop, running out of memory means swapping and a slow machine. In a container or a service with a limit, it means the process is stopped or the allocation fails, and the symptom depends on the library. To see it, I capped the address space of a process at 512 MB with ulimit -v 524288 and ran three variants on the 127 MB file:

== et-dom under a 512 MB address-space cap
object repr     : MemoryError()
lost sys.stderr
== lxml-dom under a 512 MB address-space cap
lxml.etree.XMLSyntaxError: unknown error
== lxml-iter under a 512 MB address-space cap
lxml-iter    lines=1199909 total=3001978835.00 time=4.1s peak_rss=22MB

ElementTree raised MemoryError, which at least names the problem, though the interpreter then failed while trying to report it. lxml raised XMLSyntaxError: unknown error, a message that points at the wrong cause: the file is perfectly well-formed, and the parser could not allocate. If you see "unknown error" from a parse of a big file, suspect memory before you suspect the data. The streaming variant ran to completion with the same total as before, in the same time.

Two practical points follow. First, test with a cap that matches production. A parse that works on a development machine with 32 GB is not evidence for a 512 MB container. Run the real job on the largest real file under ulimit -v or a container memory limit, and look for the failure on purpose. Second, expect the orchestrator, not Python, to report the failure in many setups: a container killed for exceeding its limit shows an exit code of 137 and no traceback, and the first thing in the log is the line that follows the last successful progress message. Log progress every so many records, so the last line before the kill tells you how far it got.

Investigating a file you have never seen

A new file from a partner arrives, 4 GB, with no documentation. Do not open it in an editor, and do not run a DOM parse to look at it. Work from the outside in, and cheaply.

Look at the beginning: head -c 2000 file.xml shows the declaration, the encoding, the root element, namespaces and the first record. Look at the end with tail -c 500, which tells you whether the file was truncated: an unfinished record or a missing closing tag means the download or the export failed, and everything you load is suspect. Count likely records with grep -c '<order ' (fast, 0.3 seconds on 127 MB, and valid when the record tag appears nowhere else). Check well-formedness with xmllint --noout --stream, which found no problems in 1.1 seconds on the test file and stays under 15 MB.

Then run a shape pass: a streaming script that counts every element path and attribute path, as in detecting structure changes in an XML feed. It takes one streaming read and tells you the record element, the depth, the optional fields and their frequency. From the counts you decide the table design. Finally, extract a sample of a few hundred records into a small file, with the same streaming parser, and do the design work on the sample in an editor or a notebook where iteration is instant. Only when the sample loads cleanly do you run the full file.

This order of operations feels slow and saves hours. Each step is bounded in time and memory, and each one answers a question that would otherwise surface as a crash at 80 percent of a long run.

Making a long job restartable

A job that takes an hour will fail at some point. The cost of a failure is the work lost. Three habits keep it small.

Load into a staging table, then swap. Write to new tables in a transaction, verify the counts and sums, and only then rename them into place. A failure leaves the old data untouched. For a feed that updates daily, this is also what lets readers keep querying during the load.

Checkpoint by record id. If the file is ordered or the records carry a monotonic id, write the last committed id after each batch. A restart skips to that id. Streaming parsers cannot seek in the middle of a file, but they can read past records quickly, which is cheaper than redoing the writes.

Make the load idempotent. Use INSERT ... ON CONFLICT or load into a temporary table and merge, so running a batch twice does not duplicate rows. This turns "did the retry double-count?" into a question with a guaranteed answer.

Add the checks from the benchmarks to the job itself: the record count in the file against the rows in the table, and one sum. They take seconds and catch silent loss. A pipeline that reports "loaded 400,000 orders, 1,199,909 lines, total 3,001,978,835.00" can be trusted, and one that reports "done" cannot.

Estimating a job before you run it

The measurements give you rates that you can scale to plan. Streaming the test file took 4.3 seconds for 127 MB, about 30 MB per second per core for a loop that touches every line. A 10 GB file at that rate is roughly six minutes on one core, and about two minutes across four workers if the work parallelizes. The same file as a DOM would need on the order of 150 GB of memory at the measured 15 times multiplier, which is why nobody parses ten gigabytes that way. These are estimates from one file shape, so treat them as the right order of magnitude and check with a run on a 1 GB slice before you commit to a schedule.

For database loads, estimate from the load rate instead: 1.6 million rows in 6.5 seconds is about a quarter of a million rows per second into two small tables with one primary key. Add secondary indexes and the rate falls, as the 2.8 times slowdown in the XML column test showed. If the table is large, create the indexes after the load. Size the disk too: the table and its indexes will usually be larger than the XML file was compressed, and smaller than it was plain, and the write-ahead log can briefly add as much again.

When the job runs on a schedule, add a margin and an alarm. A feed that grows ten percent a month doubles in seven months, and the nightly job that fit its window in January will not in August. Record the file size, the record count and the runtime of every run in a table, and chart them. The trend line tells you when to move to the next step of the decision path, long before the job fails.

A decision path

Work through these in order and stop when the problem is gone.

1. Measure a bare parse for time and peak memory on your largest real file. If both are fine, stop.

2. If memory is the problem, stream. Use iterparse with clearing, or xmllint --stream for checks. Re-measure; expect memory to drop by one to two orders of magnitude.

3. If the full job is slow and the bare parse is fast, look at the work: batch database writes, use COPY or the equivalent, remove per-record network calls.

4. If the work is still CPU bound and the file is a list of records, split and use a process pool, or process the pre-split files you already have.

5. If storage or transfer is the issue, compress, and read in place.

6. If you query the XML repeatedly, stop re-parsing it. Load it into tables, or store it in a database with indexes on the fields you ask about.

Each step is cheaper than the next, and each leaves the previous ones in place. Most files stop at step two or three.

Mistakes that look like performance problems

A few failures resemble slowness and have other causes.

A string copy of the file. etree.fromstring(open(path).read()) reads the file into a string and then parses it, holding both. Pass the path or a file object.

Parsing inside a loop. Opening and parsing the same file for every lookup turns a linear job into a quadratic one. Parse once, build a dictionary.

A slow XPath. //line searches every node, and //order[customer='x']//line inside a loop searches again for each customer. Anchor paths, or load into a database and index. XPath queries that cover most real needs shows efficient forms.

Validation on every record. Validating against a schema is much slower than parsing. Validate a sample, or validate once at intake, and parse without validation in the hot path.

Encoding surprises. A file declared as UTF-8 with Latin-1 bytes fails at an arbitrary offset, hours into a job. Check the encoding of a new feed on a sample first (XML encoding declarations and wrong encodings).

The disk. A cold first read of a 10 GB file is limited by the disk, and the same second read is limited by CPU. Time the second run when you tune.

Related reading

The posts in this cluster each take one step in depth: how much memory a DOM parser needs, benchmarking ElementTree, lxml and iterparse, streaming XML into PostgreSQL without a DOM, parallel processing of split XML files, compressing XML archives and indexing XML columns. For the shape of the data that makes these choices, read designing XML documents that age well, and for summarizing a file without a database, summarizing a large XML export by streaming.

The tools used here are documented at the lxml parsing page, the ElementTree reference, the PostgreSQL COPY command and the libxml2 xmllint manual. Look at a sample record in the XML formatter before writing the loop.

When this guide's advice stops applying

The file is small. Below a few tens of megabytes on a machine with gigabytes of memory, a tree is simpler and the cost is invisible. Do not stream a configuration file.

The format is not a list of records. A document with deep cross references, such as a CAD model or a spreadsheet's workbook XML, may need the whole tree. Increase the memory, or use a language with a lighter DOM, and measure.

Your numbers differ. These are results for one file shape on one machine. A file of long text blocks has a smaller memory multiplier, a file of tiny elements a larger one, and a different language changes the constants. The ratios held between 32 MB and 127 MB here; verify them on your own data at two sizes before extrapolating.

The real bottleneck is elsewhere. If a job spends most of its time waiting for a web service or a slow disk, none of the parsing advice helps. Profile first, with a tool, and optimize what the profile shows.

You are past the point where XML is the issue. At hundreds of gigabytes, you are in warehouse territory: land the files in object storage, convert once to Parquet and query with an engine built for it (choosing Parquet after you parse XML). Parsing XML repeatedly at that scale is a design problem, not a tuning problem.