Choosing Between XSLT XQuery and a Script for XML Jobs

There are three tools in the XML ecosystem for taking one document and producing another thing: XSLT, XQuery and an ordinary program. All three can do most jobs, and teams argue about which to use. The argument is usually conducted on taste. I ran the same job in each, so there is something to look at.

The job

For each currency in the invoices: count the invoices, count the lines, and total the revenue (quantity times amount). The answer, in every implementation that could compute the revenue, is:

EUR: 3897 invoices, 9764 lines, revenue 1,481,098.58
USD: 12138 invoices, 30245 lines, revenue 4,579,242.21
VND: 3965 invoices, 9810 lines, revenue 1,503,163.23

It is a grouping and aggregation, which is the worst case for XSLT 1.0 and an ordinary case for the others. It is not the case where XSLT shines, and I chose it to show where each tool stands at the edge of its strengths.

Four implementations

XSLT 1.0 needs the Muenchian method for grouping, and cannot compute the revenue with sum, because sum takes a node-set and the product of two fields is not a node. I wrote the counts only (grouping records with XSLT 2 and Muenchian keys):

<xsl:key name="by-cur" match="invoice" use="@cur"/>
<xsl:for-each select="invoice[generate-id() = generate-id(key('by-cur', @cur)[1])]">

XSLT 2.0 groups with for-each-group, multiplies inside the sum with a path step expression, and uses exact decimals:

<xsl:for-each-group select="invoice" group-by="@cur">
  ... sum(current-group()/line/(xs:decimal(amount) * xs:integer(@qty))) ...

XQuery is the shortest:

for $i in doc('invoices.xml')/invoices/invoice
group by $c := string($i/@cur)
order by $c
return concat($c, ': ', count($i), ' invoices, revenue ',
  format-number(sum($i/line ! (xs:decimal(amount) * xs:integer(@qty))), '#,##0.00'))

Python streams the file and accumulates in dictionaries, with Decimal for the arithmetic (summarizing a large XML export with a streaming parser):

for _, inv in etree.iterparse(path, tag="invoice"):
    for ln in inv.iterfind("line"):
        tot[inv.get("cur")] += Decimal(ln.findtext("amount")) * int(ln.get("qty"))

Results

Read the table with care. The Java tools pay about a second for JVM start-up, so for a 4.4 MB file their numbers are mostly overhead; on a 400 MB file the ratios would change, and I did not measure that. The libxslt number for the 1.0 grouping is the outlier, and I did not find out why; it says that the same stylesheet on a different engine can differ fourfold. And the Python time is for a streaming parse with the whole job in the loop, while the others build the tree first.

Everything that computed the revenue agreed to the cent. Correctness is not the axis on which these tools differ, once the arithmetic is on decimals. The 1.0 stylesheet's inability to do the revenue is the first real difference: with only floating-point numbers and no way to multiply inside a sum, it is the wrong tool for money.

Where each wins

XSLT wins when the job is restructuring a document and the document has many parts you do not want to mention. The identity template copies everything, and you override the three things that change. A 10-line stylesheet can rename elements, drop sections and wrap content in a 5,000-element document. The equivalent query rebuilds the output by hand, and the equivalent script walks the tree. XSLT also produces HTML from XML in a few templates, with escaping handled (transforming XML to HTML with XSLT), and it is built into browsers, databases, Java and .NET runtimes. Its weaknesses: 1.0 is thin (no grouping, no regular expressions, no exact numbers), the syntax is verbose, a stylesheet is hard to unit-test, and the people who can read one are few.

XQuery wins when the job is selecting and aggregating data from one or many documents and producing a result that is a list, a number or a new structure. FLWOR reads like SQL, the language is typed with exact decimals and dates, and a query can run against a database of documents with indexes (BaseX, and the XML databases of the commercial vendors). It is not good at "copy the document with small changes", where you rebuild the structure by hand. Its weakness is the ecosystem: fewer engines than XSLT, and few developers who know it.

A script wins when the job has anything else in it: reading from a database or an API, calling a service, handling errors per record, writing several outputs, running tests, logging, retrying. It also wins on large files, where streaming is natural (0.21 seconds in the table), and on team fit, since everyone can read Python or Java. It loses where the whole job is a tree rewrite that a stylesheet expresses in ten lines, and you end up walking and rebuilding the tree by hand.

A decision procedure

Ask four questions, in order.

1. Is the output the same document with changes? XSLT. Identity template plus overrides.

2. Is the output a new structure built from selected values, and is the language available? XQuery, if you have an engine and someone who reads it; otherwise a script.

3. Does the job involve I/O, state, errors per record or large files? A script.

4. Is the team the constraint? Use what they can maintain. A script that people understand beats an elegant query that one person understands.

And two practical constraints. If your runtime only has libxslt (PHP, Python's lxml, xsltproc, Ruby via Nokogiri), you have XSLT 1.0, and anything involving grouping, exact numbers or text processing should go in the host language. If you have a JVM, Saxon gives you XSLT 3.0 and XQuery for free in the Home Edition, and the picture changes: grouping and decimals are no longer reasons to leave XSLT.

A hybrid that works

The split I use most: XSLT for the document, code for the data. A script extracts and computes the numbers, hands them to the stylesheet as parameters or as a small XML document, and the stylesheet renders the HTML or the partner format. The stylesheet stays short and purely presentational, which is the part of XSLT that is good, and the logic is in code with tests. The passing-in side is covered in passing parameters into an XSLT stylesheet and running XSLT in Python with lxml.

Testing, whichever you choose

Keep an input file with the awkward cases (one member in a group, an empty group, a missing key, an ampersand, a namespace) and the expected output beside it, and run it in CI. The same fixtures work for all four implementations, and a divergence between two of them on the same file is a bug in one. For this job the expected output is the three lines above, and a change that breaks any digit is caught.

Related reading

For the basics of each see XSLT basics: templates, match and apply and XQuery basics with BaseX, and for the cluster overview XSLT and XQuery for real transformations. The languages are specified in XSLT 3.0 and XQuery 3.1, and Saxon is documented at saxonica.com. Test an XPath in the XPath tester.

When the right answer is none of the four

The data belongs in a database. If you will ask many questions of the same XML, load it into tables and use SQL.

The job is a one-liner. xmllint --xpath or xmlstarlet does it faster than any of these.

The input is not XML-shaped. Flat tabular data in XML clothing is a CSV problem. Convert it first.

The transform is a contract. If a standard defines the mapping (an invoice format conversion), use the vendor's or standard body's tool and validate the output.