Splitting a Big XML File by Repeating Element Safely

Sometimes a file is too big for the next stage. The editor chokes on it, the validator wants more memory than the machine has, the loader times out, a partner's upload limit is 10 MB. The obvious fix is to cut it into pieces, and the obvious tool is split. For XML that produces files that are not XML. The right cut is between records, and a streaming script makes it in a couple of seconds.

What split does to XML

split -l 50000 orders_nested.xml chunk_
xmllint --noout chunk_aa
xmllint --noout chunk_ab

chunk_aa:50001: parser error : Premature end of data in tag orders line 2

chunk_ab:2: parser error : Extra content at the end of the document
<order id="50000" placed="2026-01-21"><customer>c0</customer><status>new</status

split -l cuts after a number of lines. The file has the declaration on line 1, the opening <orders> on line 2 and one order per line, so the first chunk has the declaration, the opening tag and 49,999 orders, and no closing tag: "Premature end of data in tag orders". The second chunk starts with an order element and has no root at all: "Extra content at the end of the document". Five chunks were written, and none but a lucky accident of the layout could be valid. If the file were one long line, split -l would not split it at all, and split -b (by bytes) cuts anywhere, including inside a tag. csplit with a pattern for <order gets closer and has the same problem with the header and trailer. The structure has to be respected, and that takes a parser.

A streaming split

The script reads the source with iterparse, so memory stays flat, and writes each record to the current part. Every N records it closes the part with the closing root tag and opens the next with a new declaration and root.

import sys, time, resource, os
from lxml import etree

src, per = sys.argv[1], int(sys.argv[2])
os.makedirs("split_out", exist_ok=True)
t = time.time(); n = 0; part = 0; out = None

def open_part(i):
    f = open(f"split_out/part{i:03d}.xml", "wb")
    f.write(b'<?xml version="1.0" encoding="UTF-8"?>\n<orders>\n')
    return f

for _, el in etree.iterparse(src, tag="order"):
    if n % per == 0:
        if out:
            out.write(b"</orders>\n"); out.close()
        part += 1
        out = open_part(part)
    out.write(etree.tostring(el, encoding="utf-8")); out.write(b"\n")
    n += 1
    el.clear()
    while el.getprevious() is not None:
        del el.getparent()[0]
out.write(b"</orders>\n"); out.close()
print(f"{n} orders into {part} files in {time.time()-t:.2f}s, peak {resource.getrusage(resource.RUSAGE_SELF).ru_maxrss//1024} MB")

200000 orders into 8 files in 1.53s, peak 21 MB

Eight files, in 1.5 seconds, at a peak of 21 MB, from a 36.6 MB source. Each part is about 4.5 MB, the sizes ranging from 4.57 MB to 4.59 MB for the first three. The clearing idiom at the bottom of the loop is the same as in loading large XML files into a database in chunks: without it the memory would grow with the file.

Notice the details that make each part a document in its own right: its own declaration, its own opening and closing root, and the records written with etree.tostring(el), which includes the element's tail (the newline) unless you pass with_tail=False. If the source had root attributes or namespace declarations, they have to be copied into the new root on purpose. Here the root was a bare <orders>, so a literal works. For a root with a namespace, read it from the first event (iterparse with events=("start",) once) and write it into every part.

The reconciliation

A split is a file operation, and the check is counting. All eight parts must be well-formed, and the records in the parts must add up to the records in the source.

for f in split_out/*.xml; do xmllint --noout --stream $f && echo ok; done | sort | uniq -c
for f in split_out/*.xml; do xmllint --xpath "count(//order)" $f; echo; done | awk '{s+=$1} END {print s}'

      8 ok
200000

Eight parts pass, and the counts sum to 200,000, the number in the source. If the sum were off by a few, the likely cause is the boundary: a record written twice or dropped when a part was opened. If one part failed, look at the end of that file. This is the same reconciliation as after any conversion (converting XML to JSON, CSV, YAML and Excel ends with the same habit). Also check the largest and the smallest part: with a count-based split the last part holds the remainder, and it can be tiny, which some consumers dislike.

Choosing the part size

The size of a part is a property of the consumer. A loader that does well with 5,000 to 50,000 records per batch wants parts of that size; a partner whose upload limit is 10 MB wants parts under that; a validator that holds a tree wants parts that fit in memory with room to spare (15 to 20 times the part's size). Splitting by count gives parts of similar record count and different byte sizes if the records vary. Splitting by bytes means counting what you have written and closing the part when it passes a limit, which needs one more variable and gives uniform file sizes. Pick the one that matches the constraint. Record counts are easier to reconcile, and byte sizes are easier to promise to a partner.

Variations

Keep a header. Some documents have metadata before the records (<header><date>...</date></header>), which every part should repeat, or only the first. Copy it explicitly.

Group by a key, not by count. To split by customer, region or date, route each record to a part chosen by an attribute. Keep a dictionary of open files, and watch the number of simultaneously open files: a thousand regions is a thousand handles, so sort or batch.

Split a deeper level. If the repeating element is two levels down, iterparse(tag=...) still works, and the part's wrapper has to rebuild the ancestors you skip.

Command-line only. xmlstarlet sel -t -c '//order[position() <= 25000]' should extract the first 25,000 records as XML fragments; it builds a tree to do so, and I did not run it on this file. There are also dedicated utilities (the xml_split program from the Perl XML::Twig package is a well-known one). I did not have it installed here and did not test it.

Related reading

The reverse is in merging several XML files into one, and the streaming idiom behind this script in parsing large XML files with iterparse. For the family of tools, see handy command-line tools for XML. iterparse is documented in the lxml parsing guide, and split in the GNU coreutils manual. Test the paths in the XPath tester.

When splitting is the wrong fix for a big file

The next stage can stream. A streaming loader or validator handles 36.6 MB, or 3.6 GB, in flat memory. Splitting adds files to manage and a reconciliation to run.

The file is one record. A single huge element cannot be cut between records. It needs a different representation or a SAX-style handler.

Records refer to each other. Cross-references (an order pointing at a customer defined elsewhere in the file) break when records land in different parts. Keep related records together or resolve the references first.

The limit is a policy. If a partner caps uploads at 10 MB because their parser holds a tree, parts help. If it is arbitrary, ask for a streaming endpoint.