Streaming Large XML Files in Java Without Memory Trouble
Measured heap needs for 36.6 MB of XML in Java: DOM fails at 256 MB, StAX with JAXB per record runs in 16 MB, and the memory is flat whatever the file size.
Memory is the first thing that goes wrong with XML in Java. A program that works on the sample file dies with OutOfMemoryError on the real export, and the usual fix, a bigger heap, moves the failure to a bigger file. The durable fix is a different structure: stream the document and keep one record in memory at a time. This article puts numbers on the difference and shows the pattern that gives you typed objects without a tree.
The heap a tree needs
I ran the DOM program with a series of heap limits:
### DOM -Xmx128m (failed)
### DOM -Xmx256m java.lang.OutOfMemoryError: Java heap space
### DOM -Xmx512m DOM orders=200000 total=99581500.0 time=0.82s
### DOM -Xmx768m DOM orders=200000 total=99581500.0 time=0.87sA 36.6 MB file needed more than 256 MB and less than 512 MB: between 7 and 14 times the size on disk. The factor comes from how a DOM stores a document. Each element, attribute and text node is an object with headers, references to its parent and siblings, and a name. A file of short elements, like this one, is the worst case, because the per-node overhead dwarfs the content. The consequences follow directly: a 500 MB export needs a heap of 3.5 to 7 GB, and a service that parses uploads concurrently multiplies that by the number of requests in flight.
Streaming with no objects
A StAX loop that counts orders and sums totals as the events go by needs almost nothing:
### -Xmx1g StAX orders=200000 total=99581500.0 time=0.32s
### -Xmx128m StAX orders=200000 total=99581500.0 time=0.31s
### -Xmx32m StAX orders=200000 total=99581500.0 time=0.32sThe same answer and the same time at 1 GB, 128 MB and 32 MB. That is not an accident of this file. The memory a streaming parser uses is the parser's own buffers plus whatever your code keeps, and your code here keeps two numbers. Double the file and nothing changes. The code is in DOM versus SAX versus StAX parsing in Java.
Streaming with typed records
Counting is the easy case. Usually you want an object for each record: validate it, transform it, write it to a database. JAXB can bind a single element from a XMLStreamReader, and leaves the reader positioned after it, which gives the combination of both worlds:
var f = XMLInputFactory.newInstance();
f.setProperty(XMLInputFactory.SUPPORT_DTD, false);
f.setProperty(XMLInputFactory.IS_SUPPORTING_EXTERNAL_ENTITIES, false);
var u = JAXBContext.newInstance(Order.class).createUnmarshaller();
XMLStreamReader r = f.createXMLStreamReader(in);
while (r.hasNext()) {
if (r.getEventType() == XMLStreamConstants.START_ELEMENT && r.getLocalName().equals("order")) {
Order o = u.unmarshal(r, Order.class).getValue();
orders++; items += o.items.size(); sum = sum.add(o.total);
} else r.next();
}Note the else r.next(). After unmarshal(r, Order.class) the reader already sits on the event following the order end tag, so calling next() again would skip an element. The loop advances only when it did not just bind a record. This is the common mistake with this pattern: dropping every second record, with no error. The count in the output is the check.
### -Xmx512m StAX+JAXB orders=200000 items=400001 total=99581500.0 time=1.47s
### -Xmx32m StAX+JAXB orders=200000 items=400001 total=99581500.0 time=1.21s
### -Xmx16m StAX+JAXB orders=200000 items=400001 total=99581500.0 time=1.27sAll 200,000 orders and 400,001 items bound into typed objects, with an exact BigDecimal total, at heaps of 512, 32 and 16 MB. Each Order object is created, used and becomes garbage before the next, so the heap holds one record plus the collector's working space. (The JAXB classes are the ones from reading XML in Java with JAXB annotations, extended with an items list.) The time is about four times the hand-written event loop, 1.2 seconds against 0.3, which is the price of creating 600,000 objects and doing reflection-based binding. That is a good trade whenever the typed object makes the next step simpler. If the next step is only a count, skip the binding.
Putting the results side by side:
| Approach | Minimum heap that worked | Time | Result |
|---|---|---|---|
| DOM | between 256 and 512 MB | 0.82 to 1.03 s | tree in memory |
| StAX, counting events | 32 MB or less | 0.32 s | two numbers |
| StAX + JAXB per record | 16 MB | 1.2 to 1.5 s | typed objects, one at a time |
The 16 MB and 32 MB rows are the floor of what I tried, not the true minimum, and they are well below the file size.
Sizing the heap
The model is simple. Memory for a streaming job is a fixed overhead (the JVM, the parser, the classes: tens of megabytes) plus the size of the biggest record you hold, multiplied by the number of records you buffer. If you batch 1,000 orders before writing them to a database, the heap needs room for 1,000 orders. In this file one order with its items is a few hundred bytes of text and a few kilobytes as objects, so a batch of 5,000 is megabytes. For a document where a single record is large (a book with 50 MB of text in one element), no streaming parser helps with that record, and you need to treat its content as a stream too.
For a service, think in per-request terms. A DOM-based endpoint with a 10 MB upload cap needs 70 to 140 MB of heap per concurrent request. A streaming endpoint needs roughly the same few megabytes per request whatever the cap. Combine streaming with a byte cap and a time limit and the endpoint's worst case is bounded (limiting XML size and depth for uploads).
Practical details
Close the reader. XMLStreamReader.close() does not close the underlying stream; close the InputStream yourself, as the try-with-resources does above.
Do not keep the record. The point of streaming is lost if you add each Order to a list "for later". Write it out, aggregate it, or put it on a bounded queue.
Namespaces. r.getLocalName() ignores the namespace; use r.getName() (a QName) when two vocabularies share names. The JAXB class needs its namespace in @XmlRootElement.
Errors. A malformed document throws XMLStreamException with a location when the reader reaches the problem, after the earlier records have been processed. Decide what the consumer does with the records already handled: commit them, or run inside a transaction and roll back. Counting records read against the expected count catches truncation.
Security. Keep the two setProperty lines that disable DTDs and external entities, or use disallow-doctype-decl on the parser. Without them, the Java default reads files named in the document (disabling DTDs and external entities in Java and Python).
Parallelism. The reader is single-threaded. To use more cores, split the file by record into parts first (splitting a big XML file by repeating element) and process the parts in parallel, or hand the bound records to a thread pool through a bounded queue.
Related reading
For the language overview see parsing XML in Java, C# and other typed languages. The Python version of the same idiom is parsing large XML files with iterparse. The StAX API is documented in the javax.xml.stream package, JAXB's partial unmarshalling in the Jakarta XML Binding specification, and JVM heap options in the java command reference. Test an XPath in the XPath tester.
When memory is not the constraint
The files are small and few. A 200 KB configuration file in DOM is fine. Optimize for the clearest code.
You need random access. Cross-references, sorting and queries over the whole document want a tree or a database. Load into a database and query there.
CPU, not memory, is the limit. Streaming does not make parsing faster; it makes it smaller. If a job is slow, profile the code that handles each record before changing the parser.
You can use a database with XML support. Loading the file with the database's own tools may be faster and simpler than a custom loader (storing and querying XML inside a database).
