XPath Queries That Cover Most Real Needs in XML Work

XPath has a reputation for being cryptic, and the part you need is small. A short list of expressions, a few functions and a handful of traps cover nearly all the querying I have needed on XML files. This guide collects them: the expressions, the reasons they work, the places they fail quietly, and what they cost on a realistic file.

Every result was produced by running the expression on a small library document or on a 200,000-order file. The detailed articles in this group each take one topic apart; this one is the map.

What XPath is for

XPath selects nodes from an XML tree. It does not change the document, build new ones or loop. You give it an expression and it returns a node-set (or a string, a number or a boolean), and a host language, a command-line tool or XSLT does something with the result.

That makes it the right tool when the question is "which nodes?" and a clumsy one when the question is "what is the total by customer, sorted?". For the first kind it replaces dozens of lines of loop and condition code with one line. For the second, extract with XPath and compute elsewhere.

Almost every tool you will meet implements XPath 1.0: libxml2 (so xmllint, lxml, PHP's DOM, Nokogiri), browsers, and Java's built-in javax.xml.xpath. XPath 2.0, 3.0 and 3.1 add types, sequences, regular expressions, max, min, distinct-values and much more, in engines such as Saxon and BaseX. This guide uses 1.0 throughout, which is what you will probably have.

The twenty expressions

These cover what I needed. The library document is the example.

Paths

/library/shelf/book/title            a full path: titles of all books                  5
//book                               every book, any depth                              5
//book/..                            the parents of books                               2 shelves
.//title                             titles below the current node                      6
//shelf/*                            all element children of shelves                    6

Attributes

//book/@id                           attribute values                                   b1..b5
//book[@lang='vi']                   filter by attribute value                          1
//book[@available]                   has the attribute                                  5
//book[@year>=2019 and @available='true']   comparison and logic                       b1, b4

Text and children

//title/text()                       the text inside                                    6 strings
//book[title='Cache Patterns']       match on a child's text                            b1
//book[contains(title,'Notes')]      partial match                                      b2
//book[review]                       has a child                                        b1, b3, b5
//book[not(review)]                  has no such child                                  b2, b4
//book[tags/tag='redis']             any matching descendant                            b1, b4

Position

(//book)[1]                          first in the document                              b1
//shelf[1]/book[2]                   a given position in a given parent                 b2
(//book)[last()]                     last in the document                               b5

Counting and aggregating

count(//book[@available='true'])     number of matches                                  4
sum(//price[@currency='USD'])        sum of numbers                                     88.49

With | for union, normalize-space() to ignore padding, and ancestor:: or following-sibling:: for navigation, those twenty handled every question I asked. Each family has its own article: paths and basics, attribute filters, position and text predicates, functions and axes.

The traps, ranked by how often they bite

1. The default namespace

A document with xmlns="..." on the root puts every unprefixed element into that namespace. An XPath with unprefixed names then finds nothing:

//entry                  0 nodes  (Atom feed with a default namespace)
//a:entry                2 nodes  (with a: bound to the Atom URI)

This is the commonest source of "my XPath returns nothing". Bind a prefix to the URI in your host language, and put it on every element step. Attributes do not take the default namespace. local-name() works around it at the cost of speed and precision: on 100,000 entries //a:entry took 10.8 ms, //*[local-name()='entry'] 66.8 ms, and the local-name form also matched elements from other namespaces (selecting elements in a namespace with XPath).

2. Positions count within the parent

//book[1] is the first book of each parent: two results here, not one. For the first overall write (//book)[1]. The same holds for last() and position().

3. Numeric filters ignore units

//book[price>30] returned two books, and one was a 250,000 VND book. XPath compares numbers, not money. sum(//price) added dollars and dong into 250088.49. When the data has units, put the unit in the filter (//price[@currency='USD'][number(.)>30]) or convert first.

4. Existence versus value

//book[@available] keeps every book that has the attribute, including those where it is "false". //book[@available='true'] tests the value. The first reads like the second and returns 5 books instead of 4.

5. Values built into the expression

tree.xpath(f"//book[author='{evil}']/@id")      # all 5 books
tree.xpath("//book[author=$a]/@id", a=evil)     # []

String-building an expression with user input is XPath injection. Bind with variables (XPath in Python with lxml).

6. Relative versus absolute inside an element

element.xpath("//author") searches the whole document, not the element. Use .//author or author. On a book with two authors, //author returned 6 and .//author returned 2.

Two smaller traps: string(//author) returns only the first node's text, and NaN from a bad number silently fails every comparison.

What it costs

On the 200,000-order tree already parsed by lxml:

expression                                        time     result
/orders/order[status='paid']                      132 ms   66,432 nodes
//order[status='paid']                            214 ms   66,432
count(//order[status='paid'])                     183 ms   66432.0
compiled etree.XPath("//order[status='paid']")    203 ms
[o for o in iterfind('.//order') if findtext...]  251 ms
findall(".//order[status='paid']") (ElementPath)  306 ms
//order[@id='150000']                             190 ms   1
/orders/order[150000]                              43 ms   1
//order[total>400]/@id                            262 ms   39,600
sum(//order/total)                                125 ms   49,999,000

Four lessons. A full path from the root beat // by a third, because // visits every node. XPath has no indexes: an id lookup scanned the whole tree in 190 ms, so for many lookups build a dictionary once. The XPath engine beat an equivalent Python loop and ElementPath slightly, so it is not slower than doing it by hand. And parsing the file (0.44 s in lxml, 0.71 s in ElementTree) cost more than any one query.

On the command line, xmllint --xpath 'count(//order[status="paid"])' took 1.04 s because it parses and builds the tree for each call. xmllint --stream --noout parsed the same file in 0.23 s and cannot evaluate XPath. For repeated queries, load once (running XPath from the command line with xmllint).

Which tool to run it in

The Python details, including result types and variables, are in XPath in Python with lxml, and the table-building recipes in extracting values from XML into a table with XPath.

When the language runs out

XPath 1.0 has gaps that you hit sooner or later. Missing: max, min, avg, lower-case, ends-with, regular expressions, distinct-values and date functions. The workarounds, in order of preference:

Do it in the host language. Extract the values with XPath and compute in Python, SQL or JavaScript. This is the best answer for aggregates and anything involving money, since XPath numbers are floating point (the USD sum came out as 88.49000000000001).

Use the idioms. Case folding with translate(s, 'ABC...', 'abc...'). Distinct values with not(.=preceding::x), which is quadratic. Mean with sum() div count(). A suffix test with substring.

Use an extension. lxml supports EXSLT: re:test(title,'^S') matched b3 and b4. You can register a function of your own and call it from the expression.

Use a newer engine. Saxon or BaseX give XPath 2 and 3. I did not run them for this guide.

Testing expressions

An XPath is code, and it breaks when the document changes. Treat it that way. Keep a small sample document next to the code, with the cases that matter: a record with a missing child, one with a repeated child, one with an empty element, one in the wrong unit. Write the expression's expected result as a test. Add a count assertion to every extraction (len(rows) == tree.xpath("count(//book)")). And when an expression fails to match anything in production, the first things to check are the namespace and the root element's name.

When building an expression, grow it step by step: start with //shelf, check the count; add /book, check again; add the predicate. A step that returns zero nodes is the one with the problem. This is faster than debugging a long expression whole.

How XPath thinks

Most of the confusion about XPath goes away once you hold a small model in your head. An XPath expression is evaluated against a context: a current node, and in a predicate also a position and a size. A step moves from the context node along an axis and keeps the nodes that pass a name test. A predicate filters those nodes using a condition evaluated with each one as the new context. Steps are joined with slashes, and each step is applied to every node the previous step produced.

Read //shelf[@topic='databases']/book[@available='true']/title as a pipeline. Start with all shelves anywhere. Keep those whose topic attribute is databases. For each of them, take the book children. Keep the available ones. For each, take the title children. The result is the final set of nodes, in document order, with duplicates removed. Nothing in the expression loops explicitly, and nothing is evaluated in a surprising order: each step is a map-and-filter over the set before it.

Three consequences follow from the model and explain most behavior.

The result of a path is always a set, in document order, with no duplicates. If two routes reach the same node, it appears once. That is why //book/.. returned two shelves for five books.

A predicate's condition is converted to a boolean, unless it is a number, in which case it is compared with the position. That one rule explains why [1] means "first" and [@x] means "has x", and why [price] is true when a price child exists.

Comparisons between a node-set and a value are existential: author='Ana Lee' is true if any author node has that text. The same for !=, which is why author!='Ana Lee' is true for a book that has one author named Ana Lee and a second with a different name. Use not(author='Ana Lee') for "no author is Ana Lee".

Six questions, worked

The expressions in the cheat sheet are easier to keep once you have used them on questions. Here are six I asked of the library file, with the reasoning, and every answer came from running the expression.

Which available books have no reviews? The condition has two parts, a missing child and an attribute value: //book[not(review)][@available='true']/title/text(). The answer was SQL Tuning. Two predicates chained are the same as joining the tests with and. Chaining is easier to read and easier to debug, since you can remove one predicate and watch the count change.

Which authors appear more than once? XPath 1.0 has no grouping, but the preceding and following axes can express it. //author[.=following::author] selects each author who appears again later in the document, so it returned the first occurrence of each repeated name: Ana Lee and Bo Tran. The mirror image, //author[.=preceding::author], selected the later occurrences, and //author[not(.=preceding::author)] kept the first occurrence of every name, which counted 4 distinct authors among 6. The comparisons are existential, so .=following::author asks whether any later author has the same text. It is quadratic in the number of authors, which is fine here.

What is the average rating of books on the databases shelf? sum(//shelf[@topic='databases']//review/@stars) div count(//shelf[@topic='databases']//review) gave 4.0 from the reviews of 5, 3 and 4 stars. sum adds numbers from attributes, and div is the division operator (the slash is taken by paths). Without a max or avg function in 1.0, this pattern is the way. It divides by zero for a shelf with no reviews, giving NaN or Infinity, so guard it when the data may be empty.

Which USD books cost between 10 and 30? //book[price[@currency='USD'][number(.)>=10][number(.)<=30]]/title/text() returned Cache Patterns and Replication Notes. The predicate is on the price child and filters it by currency first and by range after. Putting the currency test in the same predicate as the range keeps the 250,000 VND book from sneaking into a "less than 30" or "more than 30" query. A pair of number() calls is not needed for the comparison itself, but makes the intent explicit and is harmless.

Which book has a five-star review? //review[@stars=5]/ancestor::book/title/text() found Cache Patterns. This is the case for the ancestor axis: start from the thing you can describe (a review with five stars) and walk up to the thing you want. The same answer can come from //book[review[@stars=5]]/title, which filters from the top. Both are right, and the second reads more naturally when the condition is about the children and the first when the target is the parent.

What fraction of books were published before 2019? count(//book[@year<2019]) div count(//book) returned 0.4, two of five. Arithmetic on counts is the nearest XPath comes to a report, and it is useful for sanity checks on a feed ("at least half the records should have a price").

Who comes after the librarian on the staff list? //person[@role='librarian']/following-sibling::person/text() returned Fay Ng. The sibling axes handle structures where position encodes meaning, such as a heading followed by paragraphs.

None of these needed anything beyond the twenty expressions and a few axes. Notice also how each was built in steps: first the nodes, then the condition, then the part of the result I wanted.

Writing XPath you can maintain

An expression that works today has to survive a change in the document tomorrow. A few habits help.

**Prefer explicit paths to // where the structure is known.** /library/shelf/book documents the shape and runs faster than //book: 132 ms against 214 ms on 200,000 orders. It also fails visibly (zero results) if the structure changes, where //book may start matching books in a new place (a magazine shelf, an appendix).

Name what you mean. //*[@year] matched the magazine as well as the books. //book[@year] says what you want. Avoid the wildcard unless the point is to match anything.

Keep conditions short and in the order of selectivity. Put the predicate that removes most nodes first, so later ones run on fewer. //book[@lang='vi'][contains(title,'Sao')] is cheaper than the reverse, and easier to read.

Do not compute in XPath what you can compute in code. A long expression with arithmetic and nested counts is hard to review. Select the values and compute in the host language, where you also have types and tests.

Write the namespace mapping in one place. One dictionary with the prefixes and URIs, used by all expressions in the module. When a vocabulary changes version, you change one line.

Give the expressions names. In Python, a module-level constant PAID_ORDERS = etree.XPath("/orders/order[status='paid']") documents intent and compiles once.

Test the empty case. What happens when a record has none of the child you filter on? Expressions that work on the sample often return nothing or the wrong thing for the record that lacks a field. Put such a record in the test file.

Debugging an expression that returns nothing

When an expression returns the empty set, work through the causes in order of frequency.

1. Namespace. Print the root's tag. If it has {uri} in front, bind a prefix and use it on every element step.

2. Case and spelling. XPath is case sensitive for names and string values. //Book and [@lang='EN'] find nothing here.

3. A wrong level. /library/book finds nothing because books sit under shelves. Try //book to see whether the element exists at all, then tighten it.

4. Whitespace in the value. [title='Cache Patterns'] fails if the text has a trailing space or a newline. Try normalize-space(title)='Cache Patterns'.

5. A numeric comparison with a string that does not convert. [@year>2018] with year="n/a" is false. Test with number(@year).

6. The context. Evaluating //author on an element searches the document. Evaluating author on the wrong element finds nothing.

7. The wrong document. Print the first lines of what you parsed. An HTML error page or an empty response parses and has no books.

Peel the expression back to its first step and add one piece at a time, as in the worked examples. The step at which the count drops to zero is the bug.

XPath next to other query languages

If you work with several formats, the comparisons help. CSS selectors (document.querySelectorAll('book[available=true] > title')) express a subset of XPath, only downward moves and simple filters, in a shorter syntax, and run in browsers on HTML. XPath adds the upward and sideways axes, text conditions and functions. JSONPath is modelled on XPath for JSON and shares the $.. descendant idea but not the axes (JSONPath syntax for everyday queries). jq is a filter language for JSON that is closer to a programming language. XQuery builds on XPath with loops, joins and constructors, and is how you query collections of XML documents. XSLT applies XPath expressions in templates to transform a document into another. SQL/XML functions in databases (xpath() and xmltable in PostgreSQL, value() in SQL Server) take XPath expressions as arguments, and the next group of articles covers them (storing and querying XML inside a database).

Learning XPath once pays off in all of these. The expressions in this guide work, with trivial changes, in xmllint, Python, a database function, an XSLT stylesheet and a browser's document.evaluate.

When the tree does not fit: streaming instead of XPath

XPath needs the whole document in memory. For the 28.6 MB orders file that is not a problem on a workstation, but the numbers show how it scales. Parsing the file into an lxml tree and evaluating count(//order[status='paid']) took 0.53 s in total and the process reached a peak of about 431 MB of resident memory, roughly fifteen times the size of the file. The tree is much bigger than the text, because each element, attribute and text node is an object.

The streaming alternative visits each order as the parser finishes it, tests it and throws it away:

for event, el in etree.iterparse("orders.xml", tag="order"):
    if el.findtext("status") == "paid":
        n += 1
    el.clear()
    while el.getprevious() is not None:
        del el.getparent()[0]

It found the same 66,432 paid orders in 0.85 s with lxml and 0.73 s with ElementTree's iterparse, and the memory stayed at the baseline of the process, about 21 MB, instead of 431 MB. It was a little slower than the one-shot parse-and-query, and the cost does not grow with the file. A file ten times larger would take about ten times as long and the same memory. The two cleanup lines matter: el.clear() empties the element, and deleting its earlier siblings stops the parent from holding on to the empty shells, which otherwise accumulate.

The trade is that you give up XPath across the document. In a streaming loop you test each record on its own, with findtext, find and short relative XPath on the element (el.xpath("string(status)")). Anything that needs a relation between records, such as "orders by customers who also have a refund", has to be built from accumulated state or done in a database afterwards (parsing large XML files with iterparse covers the pattern in depth).

A rule of thumb that follows: parse into a tree when the file is a few times smaller than your free memory and you want to ask several different questions; stream when it is not, or when one pass answers everything.

Putting it in a pipeline

A repeatable extraction has the same few parts every time. First, a safe parse: a parser that does not resolve entities or load DTDs, with limits on size, since XML from outside is untrusted input (safe XML parsing in Python with defusedxml). Second, a namespace map kept with the code. Third, the XPath expressions, as named, compiled constants. Fourth, conversion of the strings that come out into numbers, dates and booleans, with explicit handling of the empty case. Fifth, checks: row counts against count(//record), nulls per column against expectations, a spot check of a few records against the source. And last, a destination, whether a DataFrame, a database table or a file.

When something fails, the failure should say which expression and which record, not just "invalid". Wrap each extraction so that an exception carries the record id (o.get('id')) and the field name. A pipeline that fails on record 150,000 with "float() of empty string" is a long evening; one that fails with "order 150000: total is empty" is a short one.

And when a feed changes format, which it will, the first symptoms are zero results from an expression that used to work, a record count that falls, or a column that is suddenly all null. Alert on those, not only on exceptions. A count of matches for each main expression logged on every run is cheap and catches almost all silent changes.

What to learn first

If you are new to XPath and want the shortest path to being useful, learn in this order. Paths with / and //. Then attribute and text predicates, since filtering is the bulk of real queries. Then count, contains and normalize-space. Then the namespace rule, because it will cost you an afternoon if you meet it unprepared. Then position and the parenthesis trick. The axes can wait until the first time you need to go up or sideways, and then ancestor:: and following-sibling:: will cover it.

Practice on a document you know. Take a file from your own work, write ten questions about it in plain language, and answer each with an expression, checking the count against what you expect. After that, read the specification's list of functions once. It is short, and knowing what exists (and what does not) saves you from inventing workarounds.

Reading further

The practical reference is the list of expressions above and the spokes. For the language itself, the XPath 1.0 recommendation is short and readable, and MDN's XPath guide covers the browser side. The lxml documentation on XPath and XSLT covers variables and extensions. For structure that XPath alone cannot express, XSLT 1.0 runs on the same engine (xsltproc, lxml.etree.XSLT) and adds key(), format-number and generate-id. Understanding the document comes first: what XML is good at explains the model, and the XML formatter shows the tree.

When XPath is not the tool for the job

You are aggregating. Grouped totals, joins and sorts belong in code, SQL or XQuery.

The file is bigger than memory. XPath needs the tree. Stream, or load into a database.

The data is not XML. Use JSONPath or jq for JSON (JSONPath syntax for everyday queries), or a CSV tool for tables.

You need to change the document. XPath finds the nodes, and a library (or XSLT) edits them.