Merging Several XML Files Into One Valid Document Safely

You have a directory of XML files, one per day or one per region, and you need one file: for a loader, for a report, for a partner. The first attempt is cat. It makes a file that looks plausible and is not XML. Merging means building one document with one root from the content of several, and there are three reliable ways to do it, with different trade-offs.

Why cat fails

Each part is a document of its own:

<?xml version="1.0"?>
<orders>
  <order id="10"><customer>c1</customer></order>
  <order id="11"><customer>d1</customer></order>
</orders>

Concatenate two of them and parse:

cat parts/p1.xml parts/p2.xml | xmllint --noout -

-:6: parser error : XML declaration allowed only at the start of the document
<?xml version="1.0"?>
     ^
-:7: parser error : Extra content at the end of the document
<orders>
^

A document has one XML declaration, at the very beginning, and one root element. The concatenation has two declarations and two roots. The errors say exactly that. Stripping the declarations still leaves two roots. What you want is the children of each root inside a single new root, and that is a structural operation.

Option 1: XInclude

XInclude is a W3C standard for including parts of one document in another, and xmllint implements it. A driver file lists what to pull in:

<?xml version="1.0"?>
<orders xmlns:xi="http://www.w3.org/2001/XInclude">
  <xi:include href="parts/p1.xml" xpointer="xpointer(/orders/*)"/>
  <xi:include href="parts/p2.xml" xpointer="xpointer(/orders/*)"/>
  <xi:include href="parts/p3.xml" xpointer="xpointer(/orders/*)"/>
</orders>

xmllint --xinclude --format merge.xml

<?xml version="1.0"?>
<orders xmlns:xi="http://www.w3.org/2001/XInclude">
  <order id="10" xml:base="parts/p1.xml">
    <customer>c1</customer>
  </order>
  <order id="11" xml:base="parts/p1.xml">
    <customer>d1</customer>
  </order>
  <order id="20" xml:base="parts/p2.xml">
  ...

Six orders, in the order of the includes. The xpointer expression xpointer(/orders/*) selects the children of each part's root, so the part's own root element is not copied. Two side effects to know. First, every copied element gained an xml:base="parts/p1.xml" attribute: XInclude records where included content came from, so relative URIs inside it still resolve. That is correct and noisy, and downstream code that compares attributes or validates against a schema without xml:base will complain. xmllint has an option to suppress it, --nofixup-base-uris, when you do not need it. Second, the xmlns:xi declaration stays on the root. Neither is an error, and both make the output less clean than you might want.

XInclude is good when the list of parts is static and checked in, such as a configuration split over files. It is the wrong tool for "all the files in this directory", because the driver file has to list them, so you generate the driver with a loop, which is the same work as the next option.

Option 2: XSLT with document()

An XSLT 1.0 stylesheet can read other files with document():

<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <xsl:output indent="yes"/>
  <xsl:template match="/">
    <orders>
      <xsl:copy-of select="document('parts/p1.xml')/orders/*"/>
      <xsl:copy-of select="document('parts/p2.xml')/orders/*"/>
      <xsl:copy-of select="document('parts/p3.xml')/orders/*"/>
    </orders>
  </xsl:template>
</xsl:stylesheet>

xsltproc merge.xsl merge.xml

The output had the same six orders and no xml:base attributes, with the stylesheet's indent="yes" formatting them. The input document (merge.xml here) is only a trigger; the real data comes from the document() calls. The file names are literal in the stylesheet, so this has the same limitation as XInclude, and XSLT 1.0 has no standard way to list a directory. A common workaround is a small shell loop that builds the stylesheet or passes the file list as a parameter and calls document() for each name, which XSLT 1.0 can do only with extension functions or recursion over a string. If you reach that point, use the third option. XSLT earns its place when the merge also transforms: renaming elements, filtering, sorting or adding a wrapper in the same pass (using XSLT to transform XML into another shape).

Option 3: a few lines of lxml

import glob
from lxml import etree

root = etree.Element("orders")
for f in sorted(glob.glob("parts/p*.xml")):
    for child in etree.parse(f).getroot():
        root.append(child)
print(etree.tostring(root, pretty_print=True).decode())
print("count", len(root))

...
</orders>

count 6

Six orders. The loop is the most flexible: it handles any number of files (the glob finds them, sorted fixes the order), and you can add anything between the open and the append: skip a duplicate id, check a namespace, record which file each element came from in an attribute you choose, or catch a file that is not well-formed and continue. It is also the easiest to test. append moves the child out of its parent tree, which is fine here since each part tree is discarded.

For parts too big to hold in memory, stream: write the opening tag of the new root, then for each part iterparse its order elements and write each one out, then the closing tag. That is the reverse of the splitting script in splitting a big XML file by repeating element.

Things that go wrong in a merge

Namespaces. If the parts use different prefixes for the same namespace, the merged file has declarations repeated on copied elements. It is still correct. If parts use different namespaces for the same element names, they are not the same element, and the merged document mixes vocabularies.

Duplicates. Two parts can contain the same record, and a merge puts both in. Decide whether duplicates are an error, a "last wins" case or something to keep, and detect them: count the ids in the result against the sum of the part counts, and look for repeats. With xmllint: xmllint --xpath 'count(//order)' merged.xml should be 6, and count(//order[@id = preceding-sibling::order/@id]) gives the number of repeated ids (quadratic, so for small files).

Different structures. Parts from different versions of an export may have different optional children or element order. A schema check on the result, or on each part first, catches that (validating XML against an XSD schema).

Encoding. The parts may declare different encodings. Each is parsed by its own declaration, and the result is written in one encoding, which is what you want, as long as you read the parts as bytes and let the parser decode.

Order. glob order is arbitrary and sorted is by name. If the order of the merged records matters, make it explicit: by file date, by a field, by a key.

Headers and trailers. Parts may have summary elements (a total, a count) that are wrong after the merge. A part with <total>1000</total> merged with another does not produce a correct total by copying. Recompute.

Checking the result

Three checks, in the order I run them. Well-formedness: xmllint --noout merged.xml. Counts: the number of records equals the sum of the parts' counts (here count(//order) is 6 and each part has 2). And a spot check: pick one record from the middle of the last part and find it in the output. If a schema exists, validate the merged file against it. The counting step is the one that finds the "lost a part" and "included a part twice" bugs, and the spot check finds ordering mistakes.

Related reading

The toolbox is in handy command-line tools for XML, the reverse operation in splitting a big XML file by repeating element, and the counting recipes in counting nodes in XML from the shell. XInclude is specified in the XInclude 1.0 recommendation, and document() in the XSLT 1.0 specification. The lxml API is described in its tutorial. Test an XPath in the XPath tester.

When merging should not be done by copying elements

The parts are not records of the same kind. Merging an orders file and a customers file into one root makes a document nobody has a schema for. Keep them as separate files and join in a database.

The merged file would be enormous. A 5 GB document is hard to work with. If the consumer can read the parts, give it the parts, or a manifest listing them.

Each part is signed. Merging breaks the signatures, since the signed bytes are no longer in a standalone document. Keep the parts and a bundle, or re-sign the result.

The merge is really an update. If the parts are deltas to be applied to a base file, the operation is a patch with keys, not an append.