Parallel Processing of Split XML Files With Python Pool

An XML parser reads a document from the first byte to the last, because every byte can change the meaning of those after it. That makes a single parse a one-core job. If the file is a long list of independent records, though, you can cut it at record boundaries into several smaller documents and parse them at once. Whether that is worth doing depends on one number: how much heavier your processing is than the parsing.

Why not parse in parallel directly

You might hope to hand each worker a byte range of the file. It does not work, because a byte offset in the middle of a document has no context: the parser does not know which element it is inside, which namespaces are in scope or whether it is in a comment or a CDATA section. You could scan for record boundaries yourself, searching for the string <order and splitting there. On well-controlled data that works, and on data with the same string in a comment or a CDATA block it silently corrupts a record. A safe splitter parses once, with a real parser, and writes out valid pieces. That one pass is the cost of parallelism.

The splitter

The splitter streams records and distributes them round robin to N output files, each wrapped in its own root element:

from lxml import etree

def split(path, parts, outdir):
    outs = [open(f"{outdir}/part{i}.xml", "w") for i in range(parts)]
    for o in outs:
        o.write("<orders>\n")
    n = 0
    for _, el in etree.iterparse(path, events=("end",), tag="order"):
        outs[n % parts].write(etree.tostring(el).decode())
        outs[n % parts].write("\n")
        n += 1
        el.clear()
        while el.getprevious() is not None:
            del el.getparent()[0]
    for o in outs:
        o.write("</orders>\n")
        o.close()
    return n

Each part is a complete, valid document with the same root name as the original, so any tool that read the whole file can read a part. Round robin balances the parts without needing to know the record sizes. If records vary a lot in cost, splitting by size or using a work queue of many small parts balances better (splitting a big XML file by repeating element shows the command-line version).

One detail: etree.tostring(el) writes the element with its tail text, which here is a newline, and it keeps namespace declarations that were on ancestors. For documents with namespaces declared on the root, serialize with etree.tostring(el, with_tail=False) and add the namespace declarations to the part's root, or the pieces will repeat them on every record, which is valid and bulky.

The workers

The worker is the same extraction used in the single-process benchmark, as a function of one file:

def work(path):
    total = 0.0
    n = 0
    for _, el in etree.iterparse(path, events=("end",), tag="order"):
        for line in el.iter("line"):
            total += float(line.findtext("price")) * int(line.get("qty"))
            n += 1
        el.clear()
        while el.getprevious() is not None:
            del el.getparent()[0]
    return n, total

And the driver maps it over the parts with a process pool:

from multiprocessing import Pool
with Pool(4) as p:
    results = p.map(work, files)
print(sum(r[0] for r in results), sum(r[1] for r in results))

Results come back as small tuples, so there is almost no interprocess traffic. That is the property that makes this pattern scale: the workers read their own files and return summaries.

Results

split 400000 orders into 4 parts: 3.4s
serial   : lines=1199909 total=3001978835.00 4.1s
pool(2)  : lines=1199909 total=3001978835.00 2.1s speedup 2.0x
pool(4)  : lines=1199909 total=3001978835.00 1.1s speedup 3.8x

The scaling is close to linear: two workers halve the time, four cut it by 3.8 times. The line count and total match the single-pass run, so the split dropped and duplicated nothing. That check should be part of any splitter: count the records in the parts and compare with a count from the source.

The first line is the price. The split took 3.4 seconds, almost as long as the serial processing of 4.1 seconds. For this job, which is light per record, splitting and then parallel processing (3.4 + 1.1) takes longer than just processing once (4.1). Parallelism wins only in one of three situations:

1. The processing is much heavier than parsing. If each record needs a regex, a network call or a model, the parallel speedup applies to a larger share of the time.

2. The parts are reused. Split once, process many times, and the 3.4 seconds amortize.

3. The parts arrive split. If the provider delivers many files, or a daily file per region, there is nothing to split and the pool is free.

Processes, not threads

CPython's global interpreter lock lets only one thread run Python bytecode at a time. lxml releases the lock during the C parts of parsing, but the Python code around it, the loop and the arithmetic, holds it. For a loop like this, threads give little speedup. Processes avoid the lock at the cost of starting interpreters and copying results back, which is why multiprocessing.Pool is the right tool. (Python 3.13 has an experimental free-threaded build where threads would help, but it is not the default and libraries must support it.)

Keep the worker function at module level so the pool can pickle it, guard the driver with if __name__ == "__main__" because some platforms start workers by importing the module, and return plain data. Setting the pool size to the number of cores is a good default, and for I/O-heavy work such as database loads, a few more than the cores can be useful.

Combining with database loads

When the work is loading into PostgreSQL, each worker can run its own COPY over its own connection (streaming XML into PostgreSQL without a DOM shows the single-process loader). Four workers loading four parts into the same tables is safe for a plain insert, because rows do not conflict. Watch the database: four writers contend for I/O, and an index on the target table makes every writer slower. If the table has secondary indexes, load first and create the indexes after.

Related reading

This post belongs to parsing big XML files fast. Measurements of the parsers are in benchmarking ElementTree, lxml and iterparse, and the memory background in how much memory a DOM parser needs. The pool API is documented in the multiprocessing module reference, and the interpreter lock in the Python glossary. Check a part's structure with the XML formatter.

When parallel XML processing is not worth it

The job is I/O bound. If the work is a database call per record or a download, concurrency helps, but the right tool is batching or async I/O, not splitting files.

The file is small. Under a couple of hundred megabytes, the total runtime is seconds, and the complexity of a split costs more than it saves.

Records depend on each other. A running balance, a parent reference across records or any ordering requirement breaks round-robin splitting. Split by key so related records share a part.

You are memory limited. Four workers each hold their own parse state. With streaming that is small, with a DOM it is four times the memory.