XQuery Basics With BaseX Using FLWOR on XML Files Fast

XQuery is to XML what SQL is to tables: a query language, not a transformation language. You describe the result you want, and the processor works out how to get it. XSLT rewrites a document by rules; XQuery selects and builds. For extracting, grouping and reshaping data out of XML files, especially when the answer is a list or a number, XQuery is often shorter than the equivalent stylesheet. This article runs the basics on BaseX, a free XQuery processor and XML database.

Running a query

BaseX is a Java program with a command-line mode, a REPL, a GUI and an HTTP server. The one-shot form takes a query as an argument and the input document with -i:

java -cp "lib/*" org.basex.BaseX -i library.xml 'count(//book)'

In the query, //book is XPath, and everything XPath can select, XQuery can use. The language is a superset: any XPath 2.0 or 3.1 expression is a valid query.

FLWOR

The central expression is FLWOR (said "flower"): for, let, where, order by, return. It binds variables, filters, sorts and builds a result.

for $b in //book
let $p := xs:decimal($b/price)
order by $p descending
return concat($b/@id, ' | ', $b/title, ' | ', string-join($b/author, '; '), ' | ', $p, ' ', $b/price/@currency)

b3 | Indexes "Explained" | Di Park | 250000 VND
b2 | SQL Tuning, 2nd Edition | Cy Ortiz | 44 USD
b1 | Cache Patterns | Ana Lee; Bo Chen | 29.5 USD

for $b in //book iterates; let $p := ... binds a value once per book; order by $p descending sorts; return builds one result per iteration. The result is a sequence of three strings. Two details. The price printed as 44 and 29.5: casting to xs:decimal gives a number with an exact value, and its canonical text drops trailing zeros, so 44.00 becomes 44. If you need the original formatting, return the text ($b/price), or format with format-number. And string-join($b/author, '; ') collected the authors of each book, whether there are one or two. That is the sort of operation that takes a recursive template or a loop with a position test in XSLT 1.0.

Because the sort key is xs:decimal, the order is numeric, not textual, which avoids the trap that XSLT's default text sort sets (sorting and filtering XML with XSLT). XQuery is typed, and when it sorts numbers it knows they are numbers.

Grouping

for $b in //book
group by $c := string($b/price/@currency)
order by $c
return concat($c, ': ', count($b), ' book(s), total ', sum($b/price ! xs:decimal(.)))

USD: 2 book(s), total 73.5
VND: 1 book(s), total 250000

group by is a clause of XQuery 3.0 and later. After it, the variable $b stands for the sequence of books in the group, so count($b) is the group size and $b/price the group's prices. The ! operator maps an expression over each item, here converting each price to a decimal before sum. The USD total is 73.5, exact. The same query shape on the real data:

let $d := doc('invoices.xml')
for $i in $d/invoices/invoice
group by $c := string($i/@cur)
order by $c
return concat($c, ': ', count($i), ' invoices, revenue ', format-number(sum($i/line ! (xs:decimal(amount) * xs:integer(@qty))), '#,##0.00'))

EUR: 3897 invoices, revenue 1,481,098.58
USD: 12138 invoices, revenue 4,579,242.21
VND: 3965 invoices, revenue 1,503,163.23

The revenue matches, to the cent, the figures from the Python streaming script, the DuckDB query and the XSLT 2.0 stylesheet earlier in this series. The query took 1.30 seconds from start to finish including the start-up of the JVM and BaseX and parsing a 4.4 MB file. Compare the Muenchian XSLT 1.0 grouping at 4.1 seconds on libxslt and a Python loop at 0.03 seconds on a tree already in memory (grouping records with XSLT 2 and Muenchian keys). Read those as orders of magnitude, not benchmarks: JVM start-up dominates a file this small.

Constructing XML

XQuery can also build documents. Curly braces inside an element constructor switch back to expression mode:

<catalog count="{count(//book)}">{
  for $b in //book[price/@currency = 'USD']
  return <item sku="{$b/@id}"><name>{string($b/title)}</name><note>{if ($b/note) then string($b/note) else 'none'}</note></item>
}</catalog>

<catalog count="3"><item sku="b1"><name>Cache Patterns</name><note>Covers R&amp;D</note></item><item sku="b2"><name>SQL Tuning, 2nd Edition</name><note>none</note></item></catalog>

The count attribute is computed (the total of all books, 3, not just the selected two: the expression is evaluated over the whole document, which a careless reader might not expect). Each item takes its sku from the id and its name from the title. The conditional if ... then ... else supplies a default for the book without a note, and the ampersand in the first note was escaped on output. This is the XQuery equivalent of an XSLT template, with the content and the logic in one expression. For a document-to-document transformation with lots of structure to preserve, XSLT's identity-plus-overrides is usually shorter; for building a new, different structure from selected data, XQuery reads better.

Where BaseX goes beyond a file

BaseX is also a database. CREATE DB orders orders.xml loads a document into an indexed store, after which queries use db:get('orders') or just run in the context of the database, and indexes on text and attributes make selective queries fast on files far larger than memory. The HTTP server exposes REST and a query endpoint. I used only the one-shot mode on files here, and did not test the database mode or its indexes, so I will not quote performance for them. The relevant point is the shape of the growth path: the same query that runs on a file runs on a database.

Typing is the difference

Two properties of XQuery show up in every query above. It is typed: xs:decimal, xs:integer, xs:date are real types with exact arithmetic and correct comparison, and a cast of text that is not a number raises an error rather than producing NaN. And it is declarative over sequences: every expression produces a sequence, and the operators (!, /, ,) map and flatten them. Both reduce the code needed for the data-shaped tasks that XML is used for, and both can surprise people coming from XSLT 1.0 or scripting languages, where values are strings and numbers are doubles.

Related reading

For the stylesheet alternative see XSLT basics: templates, match and apply and using XSLT to transform XML into another shape. For the comparison, choosing between XSLT, XQuery and a script. The language is specified in XQuery 3.1, and BaseX is documented at docs.basex.org. Test an XPath, the starting point of every query, in the XPath tester.

When XQuery is more than the job needs

One value from one file. xmllint --xpath answers it faster to type and to run.

A heavy transformation of a document. Identity-plus-overrides in XSLT preserves everything you do not mention. In XQuery you rebuild what you want, which is more code for a document with many parts.

The team has a Python or Java codebase. A query in a separate language, run by a separate tool, is one more thing to maintain. Use it where its brevity pays.

The data is not XML. BaseX also reads JSON and CSV, and SQL is a better home for tabular data.