Compressing XML Archives and Reading Them In Place

XML is verbose, and the verbosity is highly repetitive: the same element and attribute names, again and again, for every record. That makes it one of the better formats to compress. Archives of daily feeds, exports and dumps are almost always stored compressed, and the question is which compressor to pick and whether you have to unpack the file to read it. The answers are measurable and shorter than expected.

Sizes and times

Compress the file with each tool at a common setting and read the sizes:

gzip -6 -c o400k.xml > o400k.xml.gz
zstd -3 -q -f -c o400k.xml > o400k.xml.zst
xz -6 -T4 -c o400k.xml > o400k.xml.xz

Every tool took the file to about a tenth of its size or less. Zstd at level 3 matched gzip's ratio in a sixth of the time. Xz saved a further 4 MB, 27 percent smaller than gzip, and paid for it with 16 times gzip's wall time. These numbers describe this file, which is highly regular (500 distinct customers, a few thousand SKUs, fixed structure). Real feeds with more text compress somewhat less, and the ordering of the three holds.

The practical advice is short. Use gzip when the consumer has to be able to open it anywhere: every language reads gzip from its standard library, every browser and HTTP client handles Content-Encoding: gzip. Use zstd when you control both ends and care about speed, since it is much faster at similar ratios. Use xz for cold archives that are written once and read rarely, where 24 seconds is cheap and the saved space is permanent.

Reading without unpacking

You do not need to decompress to a file first. The decompressor can feed the parser directly. In Python, gzip.open gives a file object that decompresses as you read, and iterparse accepts any file object:

import gzip
from lxml import etree

with gzip.open("o400k.xml.gz", "rb") as f:
    for _, el in etree.iterparse(f, events=("end",), tag="order"):
        handle(el)
        el.clear()
        while el.getprevious() is not None:
            del el.getparent()[0]

I ran the same summing job as the earlier benchmarks on the plain file and on the gzip file:

lxml-iter    lines=1199909 total=3001978835.00 time=4.2s peak_rss=22MB
lxml-iter-gz lines=1199909 total=3001978835.00 time=4.3s peak_rss=22MB

Reading the compressed file cost 0.1 seconds more and the same 22 MB of memory. The parser consumes the decompressed stream in blocks and never holds the 132 MB. Open the file in binary mode ("rb") so the parser sees the bytes and can read the encoding declaration itself. For zstd, use the zstandard package's ZstdDecompressor().stream_reader(f), and for xz, lzma.open, both of which present a file object the same way.

On the command line, decompress to a pipe:

zcat o400k.xml.gz | python3 job.py

That ran in 2.5 seconds, faster than reading the gzip inside Python (4.3 seconds). The reason is parallelism: zcat is a separate process, so decompression runs on one core while parsing runs on another, where gzip.open does both on one. If your job reads standard input (etree.iterparse(sys.stdin.buffer, ...)), a pipe from zcat, zstdcat or xzcat is a free speedup for large files.

Compression does not shrink memory

A compressed file is smaller on disk and on the wire. It does not change what the parser holds once the data is read. A DOM parser decompresses the content and builds the same tree, needing the same ten to fifteen times the uncompressed size (how much memory a DOM parser needs). The savings are in storage and transfer, and streaming is what saves memory.

It also does not make random access possible. A gzip stream can only be read from the start; to jump to the middle you decompress everything before it. If you need to read parts of a large archive independently, compress in independent blocks: one file per day or per region, or a format with a block index (zstd has a seekable variant, bgzip for gzip-compatible files with an index). Splitting before compressing also helps parallel processing (parallel processing of split XML files).

Archives of many files

Two layouts are common. A tar of XML files compressed as a whole (.tar.gz) compresses best, because the compressor sees the repetition across files, and can only be read sequentially. A zip compresses each file separately, which costs some ratio and allows reading one member without touching the others. Office documents are zips of XML for that reason (reading Office Open XML (docx) opens one). In Python, zipfile.ZipFile(path).open("name.xml") returns a file object you can hand to iterparse the same way.

For downloads, check whether the server sends the file compressed and your client is asking for it. Many feeds are served as .xml.gz, and a few as plain .xml with Content-Encoding: gzip, which HTTP libraries decompress for you (handling compressed XML downloads goes through the cases).

Picking a setting for your pipeline

The levels matter less than the tool. Gzip level 6 is the default for a reason: higher levels cost noticeably more time for a few percent of size. Zstd's levels run from 1 to 19, with level 3 as the default, and the high levels approach xz's ratio at much higher cost, so test your own data at two or three levels before settling. If you compress daily files that the next job reads straight away, optimize for speed, and if you keep them for a year, optimize for size.

Record what you did. A file named feed-2026-10-27.xml.zst tells the next reader what tool to use, and a manifest with the compressed size, the original size and a checksum of the uncompressed content tells them whether the file survived the transfer. Verify the checksum of the uncompressed stream, since a compressed file can be intact while the content inside is not what the producer wrote.

Related reading

This post belongs to parsing big XML files fast. To sum a compressed file's records without a database, see summarizing a large XML export by streaming. The tools are documented in the zstd manual and the Python gzip module reference. Look at a decompressed sample in the XML formatter.

When compressed XML needs a different approach

You need random access. Single-stream gzip, zstd and xz files must be read from the start. Use independent blocks, an indexed format, or a database.

The data is already compressed inside. XML with embedded Base64 images will not shrink much, because the image bytes are already compressed (embedding binary data in XML).

The consumer cannot decompress. A partner who asks for plain XML gets plain XML, and the transport (HTTPS with Content-Encoding: gzip) can compress on the wire.

Security matters. A tiny compressed file can expand to gigabytes (a decompression bomb). Limit the output size when you read untrusted archives (limiting XML size and depth for uploads).