Limiting XML Size and Depth for Uploads With Measured Limits

An upload endpoint that accepts XML is an endpoint that accepts whatever size and shape the sender likes, unless you say otherwise. The attacks in this cluster (entity expansion, external entities) have specific fixes. A third family needs no tricks: a document that is too big, too deep or too wide for the machine that parses it. Limits are the answer, and the useful question is what numbers to set. I measured where the libraries stop on their own and what a document costs, so the limits can come from data.

Depth

A document nested a hundred thousand levels deep is a few hundred kilobytes and can crash a recursive consumer. I generated <a> repeated n times around a character and parsed it three ways:

depth    200 lxml            ok 0.000s
depth    255 lxml            ok 0.000s
depth    257 lxml            XMLSyntaxError: Excessive depth in document: 256, use XML_PARSE_HUGE option
depth    257 lxml huge_tree  ok 0.000s
depth   5000 lxml            XMLSyntaxError: Excessive depth in document: 256, use XML_PARSE_HUGE option
depth   5000 lxml huge_tree  XMLSyntaxError: Excessive depth in document: 2048, use XML_PARSE_HUGE option
depth   5000 ElementTree     ok 0.001s
depth 100000 ElementTree     ok 0.039s

lxml has a default depth limit of 256 and stopped at 257 with an error that names the option that raises it. With huge_tree=True the limit rises to 2,048, a ceiling that is not infinite, and the 5,000-level document was still refused. ElementTree has no depth limit at all: it parsed 100,000 levels in 0.04 seconds. So whether an adversarial depth is refused depends on the library, and the standard one does not refuse it.

Depth matters later more than at the parse. Any code that walks the tree recursively (a pretty printer, a converter to a dictionary, a schema validator in pure Python) uses the call stack in proportion to the depth, and Python's default recursion limit is about a thousand frames. A 100,000-level document that ElementTree accepted would crash a recursive walker with RecursionError, which is a denial of service by a 200 KB upload. The fix is a depth check of your own, with a streaming pass:

def max_depth(path_or_file, limit=100):
    depth = 0
    for ev, el in etree.iterparse(path_or_file, events=("start", "end")):
        depth += 1 if ev == "start" else -1
        if depth > limit:
            raise ValueError("document too deep")
        if ev == "end":
            el.clear()

Real data is shallow. The deepest document in the other articles in this cluster had four levels, and document formats such as DocBook reach perhaps twenty. A limit of 50 to 100 rejects nothing legitimate and everything adversarial.

Text nodes

A single huge text node is the other single-element trick. lxml limits one to 10 MB unless told otherwise:

  5 MB text node default    ok 0.01s
  5 MB text node huge_tree  ok 0.00s
 11 MB text node default    Resource limit exceeded: Text node too long, try XML_PARSE_H...
 11 MB text node huge_tree  ok 0.01s
 60 MB text node default    Resource limit exceeded: Text node too long, try XML_PARSE_H...
 60 MB text node huge_tree  ok 0.07s

A 5 MB node parsed, an 11 MB one was refused with a message that again names the option, and huge_tree=True accepted 60 MB in seven hundredths of a second. The default is a useful guard against a long string where records should be, and also refuses legitimate documents that embed a large payload (a base64 image in an element, a long log). That is the case where people reach for huge_tree=True and switch off all the limits at once. If you must accept big embedded payloads, a better design is an upload of the file itself, with the XML referencing it.

Width

Depth and text are not the only axes. A flat file with two million small elements:

2,000,000 siblings 2000000 0.27s peak MB 329

Parsed in 0.27 seconds, with a peak of 329 MB. The document is 8 MB (<b/> is four bytes), so that is 40 times the file size, more than the 15 to 20 that records with text and attributes cost. Empty elements are the worst case per byte, because each one is an object. No depth or text limit would have flagged it.

What a document costs

Across the measurements in this cluster, a parsed tree takes 15 to 40 times the size of the file, with the high end for many tiny elements:

This gives the sizing rule for an endpoint: peak memory per request is about 20 to 40 times the byte cap. A cap of 20 MB means 400 to 800 MB for one request, and with ten concurrent uploads that is up to 8 GB. If the cap is 1 MB, the same ten requests cost 400 MB. So the byte cap has to be chosen with the concurrency in view, and the ratio is the reason a streaming parser (iterparse) is worth the extra code for any endpoint that accepts large files: its memory stays flat at 20 to 70 MB whatever the size (loading large XML files into a database in chunks).

Setting the limits

For an upload endpoint, set them in this order, from the cheapest check to the most expensive.

1. Request size. Reject by Content-Length before reading, and enforce it again while reading, since the header can lie and chunked uploads have none. Read at most the cap plus one byte and refuse if you get it all. In a web server, set the maximum body size for the route (client_max_body_size in nginx) so the application never sees the oversized request.

2. Content type and a quick look. Check the declared type, and reject a document that starts with a DOCTYPE if you do not need DTDs (disabling DTDs and external entities in Java and Python).

3. Depth and element count in a streaming pre-pass, or in the parser's own options (lxml defaults, or SAX handlers that count). A cap of 100 levels and a million elements is generous for data exchange.

4. Text and attribute length. Keep the parser's defaults; do not set huge_tree=True for untrusted input.

5. Time and memory at the process level. Run the parse in a worker with a memory limit and a timeout, so that a case your numbers missed kills one job.

6. Rate. Per-client request limits, because every other limit is per request.

Tell the sender what failed. A 413 with "document larger than 5 MB" or a 400 with "nesting deeper than 100 levels" is better than a timeout, and costs nothing to implement. Log the rejections with the source, the same as the amplification guard, because a stream of them is a signal.

Choosing the numbers

Start from the largest legitimate document you have seen, add a margin, and cap there. For the feeds in this series, the largest file was 36.6 MB (a batch export) and the typical upload was well under 1 MB. A synchronous API for single orders can cap at 1 MB; a batch import endpoint can cap at 100 MB, accept it as a file, and process it asynchronously with a streaming parser. Different endpoints get different limits. Treat the numbers as configuration, with a test per endpoint that uploads a document just over the cap and checks for the rejection.

Related reading

For the overview see XML security problems to close before production, and for the entity attacks stopping the billion laughs attack in your parser. The streaming approach is in parsing large XML files with iterparse. The libxml2 limits are described in the libxml2 parser options, and the OWASP view in the XML External Entity Prevention Cheat Sheet.

When limits are the wrong control

The sender is trusted and the files are big on purpose. A nightly export from your own system can be 2 GB. Process it with a streaming parser and an offline job, and do not cap it as if it were a web upload.

The danger is not size. A 2 KB document can carry an external entity or a signature-wrapping trick. Limits are one layer.

Your library enforces its own limits. If the parser's defaults are already safe (as lxml's depth and text limits are), adding code that does the same is redundant. Add limits for what the library does not cover, such as ElementTree's depth.

The limit would reject real data. A legitimate document with a 15 MB embedded payload fails a 10 MB text guard. Change the design (reference the payload), not the guard.