Handling Compressed XML Downloads Without Surprises

Large XML is usually compressed, and the compression arrives in more than one way. A public data set is a .xml.gz file. A web server compresses a response on the fly and says so in a header. A badly configured server sends compressed bytes with no label at all. The three look the same in a browser and different in code, and the difference is the cause of a regular bug: "I got a file that is not XML".

Three ways gzip arrives

I set up three routes on the test server, each returning the same small Atom feed gzipped:

Fetching each with two clients:

/atom.xml.gz   requests:  288 bytes starts 1f8b0800 | urllib:  288 bytes starts 1f8b0800 | Content-Encoding=None type=application/gzip
/atom-ce.xml   requests:  688 bytes starts 3c3f786d | urllib:  288 bytes starts 1f8b0800 | Content-Encoding=gzip type=application/xml
/mislabel.xml  requests:  288 bytes starts 1f8b0800 | urllib:  288 bytes starts 1f8b0800 | Content-Encoding=None type=application/xml

The bytes after the "starts" tell the story: 1f8b is the gzip signature, 3c3f786d is <?xm, the start of an XML declaration. For the middle route, requests returned 688 bytes of XML: it saw Content-Encoding: gzip and decompressed. urllib returned 288 bytes of gzip, since it hands you the body as sent and expects you to deal with the header. Code written against one client and copied to the other fails in this exact way. The first and third rows are the same in both: the bytes are gzip, and the client does not know or care. The first is a file you asked for. The third is a server bug (it compressed and forgot to say), and you meet it in practice more often than you would like.

Transfer-encoded gzip is controlled by the request as well: a client that sends Accept-Encoding: gzip invites the server to compress, and requests sends it by default; urllib does not. If you want smaller transfers from urllib, add the header and decompress yourself.

Detect by content, not by name

The dependable answer is to look at the first bytes:

def load_xml(data: bytes):
    if data[:2] == b"\x1f\x8b": data = gzip.decompress(data)
    elif data[:3] == b"BZh": data = bz2.decompress(data)
    elif data[:4] == b"PK\x03\x04":
        z = zipfile.ZipFile(io.BytesIO(data)); data = z.read(z.namelist()[0])
    return etree.fromstring(data)

/atom.xml.gz -> 7 children via magic-byte sniffing
/atom-ce.xml -> 7 children via magic-byte sniffing
/mislabel.xml -> 7 children via magic-byte sniffing

All three routes parse to the same seven children. The signatures are stable: gzip starts 1f 8b, bzip2 BZh, zip PK 03 04 (and zip may hold several files, so pick by name when you can). A file name or a Content-Type is a claim and the bytes are the evidence. The XML parser still receives bytes, not a decoded string, so the declared encoding is honored (encoding mismatch errors in XML documents).

Sizes: what compression buys

The 36.6 MB orders file compressed as follows:

36.6 MB file: gzip 3.4 MB (10.8x, 0.2s), bzip2 1.4 MB (26.6x)

gzip gave a 10.8 times reduction in 0.2 seconds, bzip2 26.6 times and slower. XML is repetitive (tag names repeat in every record), so it compresses far better than most data. The practical consequence is that the transfer is cheap, and that the stored .gz is what you keep: compare the compressed size with the source system's reported size to check that a download is complete, and keep the checksum of the compressed file, since that is what the publisher usually lists.

Streaming from a compressed file

Do not decompress to disk, and do not read the whole file into memory. gzip.open returns a file object, and iterparse reads from file objects:

with gzip.open("orders.xml.gz", "rb") as f:
    for _, el in etree.iterparse(f, tag="order"):
        n += 1; el.clear()
        while el.getprevious() is not None: del el.getparent()[0]

stream from .gz: 200000 orders in 0.67s, peak 21 MB

200,000 orders in 0.67 seconds at a peak of 21 MB, from a 3.4 MB file that expands to 36.6 MB. Decompression adds little. (My first run of this reported 87 MB, because I had left the raw file and both compressed copies in memory in the same process; the streaming loop itself needed 21 MB. Measure a streaming job in its own process.) The same works over HTTP: requests.get(url, stream=True) and iterparse(r.raw), with r.raw.decode_content = True when the server uses transfer encoding, to stream from the network without a temporary file. I tested the file case, not the network streaming case.

Compression bombs

Compression makes size an unreliable guide. A 200 MB file of one repeated character compressed to 873 KB, a ratio of 230 to 1, and a deliberately crafted file can do far better. A downloaded .gz of a few kilobytes can expand to gigabytes, and an XML parser or gzip.decompress that reads it all into memory will oblige. The guard is to cap what you read from the decompressor, not what you receive:

def bounded(data, limit):
    d = gzip.GzipFile(fileobj=io.BytesIO(data)); out = d.read(limit + 1)
    if len(out) > limit: raise ValueError(f"decompressed size exceeds {limit} bytes")
    return out

bomb compressed 872775 bytes -> expands to ~200 MB
guard: decompressed size exceeds 50000000 bytes

Reading one byte past the limit and failing is enough. For a streaming parse, wrap the file object in a reader that counts bytes and raises past the cap. Apply the same idea to zip archives (check each member's declared size and the total, and never use the names in the archive as paths without sanitizing them) and to HTTP responses with transfer encoding, where the client library decompresses for you: a size cap on the decoded body is a setting some clients do not have. The XML-level limits (entities, depth, text size) still apply after decompression (limiting XML size and depth for uploads).

Habits

Fetch with a timeout and check the status before looking at bytes; an error page is HTML, not gzip.

Sniff, decompress, parse as separate steps, so the failure message says which one failed: "not gzip", "gzip truncated", "not well-formed".

Handle truncated downloads. A gzip stream cut short raises EOFError: Compressed file ended before the end-of-stream marker was reached when read to the end, and an XML parse of the partial output raises a syntax error. Both mean the file is incomplete. Retry the download; do not process the part.

Keep the compressed file and its checksum with the loaded data.

Prefer compressed storage. At a ratio of 10 or better, XML archives cost little to keep.

Related reading

For the incremental fetch see incremental XML feed imports by last modified, and for the overview working with XML feeds and APIs. The streaming pattern is in parsing large XML files with iterparse. The gzip format is RFC 1952, and content codings are in RFC 9110. Test a path in the XPath tester.

When the compression is not the real problem

The file is not compressed, it is wrapped. An archive with a manifest, several XML files and attachments is a package; read the manifest.

The XML inside is the issue. After decompression the same checks apply: encoding, well-formedness, size, depth.

The server is the issue. Wrong headers and truncated transfers are the publisher's bugs. Report them with the failing request, and add the workaround with a comment.

You can ask for another format. A paginated API returning small uncompressed responses is easier on every client than a nightly 2 GB archive.