XPath Basics Selecting Nodes by Path in XML Documents

XPath is the query language for XML. If you can read a file path, you can read an XPath: /library/shelf/book/title walks down from the root, one step per slash. The expression language has more in it than paths, and a small subset covers most work. This article is the subset, tried on a small document so that every result is visible.

The document

<library name="City Library" opened="1998">
  <shelf id="s1" topic="databases">
    <book id="b1" year="2019" lang="en" available="true">
      <title>Cache Patterns</title>
      <author>Ana Lee</author>
      <price currency="USD">12.50</price>
      <tags><tag>cache</tag><tag>redis</tag></tags>
      <review stars="5">Clear and <em>practical</em>.</review>
      <review stars="3">Too short.</review>
    </book>
    ... (b2, b3)
  </shelf>
  <shelf id="s2" topic="formats">
    ... (b4, b5)
    <magazine id="m1" year="2023"><title>Weekly Data</title>...</magazine>
  </shelf>
  <staff><person role="librarian">Eve Cho</person><person role="clerk">Fay Ng</person></staff>
</library>

Think of it as a tree: library is the root, shelf and staff are its children, book and magazine are children of shelves, and so on. XPath moves around that tree.

Steps and slashes

A path is a sequence of steps separated by slashes. Each step selects nodes relative to the ones before it.

/library                         1 node    the root element
/library/shelf                   2 nodes   s1, s2
/library/shelf/book/title        5 nodes   the title of every book

A leading slash starts at the root of the document. Each following / goes down one level to the children with the next name. If there is no element with that name at that level, the result is empty, and there is no error. /library/book would match nothing, since the books are not direct children of the library.

The double slash is the shortcut for "at any depth":

//book                           5 nodes   b1 to b5, wherever they are
//title/text()                   6 strings the 5 book titles and the magazine's
//book/..                         2 nodes   the parents: s1 and s2
.//title                          6 nodes   from the current node down, any depth

//book is the most used expression there is, and the most overused, since it searches the whole document. //title/text() returned 6 strings, not 5, because the magazine also has a title. A path that you expected to select only books can include other things with the same name. Constrain it by writing the full path when you know it (/library/shelf/book/title gave 5).

Attributes, text and wildcards

Attributes use @:

//book/@id                       5 nodes   b1 to b5 (attribute nodes)
/library/@name                   1 node    'City Library'
//*[@year]                       6 nodes   anything with a year: the 5 books and the magazine

The text inside an element is a node of its own, selected with text():

//title/text()                   'Cache Patterns', 'Replication Notes', ...
string(//book[1]/title)          'Cache Patterns'

text() returns the text children, and string() returns the concatenated text of an element and everything below it. For <review>Clear and <em>practical</em>.</review>, text() gives two pieces, 'Clear and ' and '.', and string() gives 'Clear and practical.' (why whitespace and mixed content matter).

The wildcard * matches any element, and node() matches any node at all:

//shelf/*                        6 nodes   all element children of the shelves
//shelf/node()                  14 nodes   the same elements plus the whitespace text between them

//shelf/* returned 6 elements, and //shelf/node() returned 14: six elements and eight text nodes that are the indentation. This is why a path with node() or text() returns more than you can see.

Results come in four kinds

An XPath expression evaluates to one of four types, and tools hand them back differently.

//book                           a node-set (a list of nodes)
string(//book[1]/title)          a string      'Cache Patterns'
count(//book)                    a number      5.0
boolean(//book[@id='b9'])        a boolean     False

A node-set is what you usually want. Functions turn it into one of the others. In lxml, tree.xpath(...) returns a list for node-sets, and a float, a string or a bool otherwise (XPath in Python with lxml). Counts are floats in XPath 1.0 (there is only one numeric type), so count(//book) came back as 5.0.

The position trap

The first surprise for most people is positions:

//book[1]                        2 nodes   b1 and b4
(//book)[1]                      1 node    b1
//book[last()]/@id               'b3', 'b5'
(//book)[last()]/title/text()    'Pocket Regex'

//book[1] does not mean the first book in the document. It means every book that is the first book child of its parent. There are two shelves, so there are two results. To get the first book overall, parenthesize the whole path and then index it: (//book)[1]. The same applies to last() and position() (using XPath predicates for position and text).

The remaining steps

A few more pieces complete the basics.

. is the current node, and .. is its parent. //book/.. returned the two shelves. They let you move back up after filtering.

| joins two node-sets: //book[1]/title | //book[1]/author.

Whitespace in an expression is free, and names are case sensitive (//Book finds nothing).

An absolute path (starting with /) begins at the document root, even if you evaluate it from an element. A relative path (author/text()) begins at the current node. Evaluating //author from inside one book still searched the whole document and returned all 6 authors, which is a common bug when you meant "this book's authors". Use .//author or author.

A worked reading of one expression

Take //shelf[@id='s2']/book[@available='true']/title/text() and read it left to right. //shelf finds every shelf anywhere. [@id='s2'] keeps the one whose id attribute is s2. /book moves to its book children. [@available='true'] keeps the available ones. /title moves to their titles, and /text() takes the text inside. The result is SQL Tuning and Pocket Regex. Every XPath you meet is a longer version of that: a sequence of moves, each followed by an optional filter.

When an expression returns something unexpected, cut it back to a prefix and evaluate again. //shelf[@id='s2'] should return one node; adding /book should return two; the filter should keep two, and so on. A step that returns zero nodes is the one to look at. This peeling approach finds most mistakes faster than staring at the whole expression, and the XPath tester makes each step a few seconds.

What to use it for

For quick extraction, XPath replaces dozens of lines of loop code. A single expression //book[@available='true']/title/text() does what a nested loop with conditions would. It is also portable: the same expressions work in xmllint, Python, Java, PHP, XSLT and most databases that store XML. The next articles in this group cover filtering by attribute (filtering XML nodes by attribute with XPath), predicates, functions, namespaces, the command line and axes. Try expressions interactively in the XPath tester and look at the structure in the XML formatter. The language is defined in the XPath 1.0 recommendation, and the tutorial that most people learn from is the MDN XPath guide.

For documents that use namespaces, the paths above find nothing until you map a prefix (how XML namespaces work). The cluster overview is XPath queries that cover most real needs.

When a path is not the right tool

You need to read everything. If you will visit every element anyway, a loop over the tree is as clear and often faster than many separate queries.

The file is very large. XPath needs the tree in memory. For files larger than memory, stream with iterparse and test each element (parsing large XML files with iterparse).

The query logic is complex. Joins across elements, grouping and arithmetic over many nodes are easier in code, in XSLT, or in a database after loading the data.

The data is in a different format. For JSON, use JSONPath, jq or a database (JSONPath syntax for everyday queries). XPath only reads XML.