Extracting Values With xmlstarlet From the Shell Fast

xmllint --xpath gives you one value per run, which is awkward as soon as you want a column. xmlstarlet fills the gap. It is a command-line toolkit that selects, edits, validates and transforms XML, and its sel command runs XPath with a small template language that can print several values per record. When I need a quick list out of a file and do not want to open a script, it is what I use.

The template language

The sel command takes -t to start a template, and inside it a few options build the output:

Put together, a template walks the file the way you would write a loop. Titles of the books:

xmlstarlet sel -t -m '//book' -v 'title' -n library.xml

Cache Patterns
SQL Tuning, 2nd Edition
Indexes "Explained"

-m '//book' loops over each book, -v 'title' prints the title relative to the current book, and -n ends the line. Several values per line, with separators:

xmlstarlet sel -t -m '//book' -v '@id' -o ',' -v 'price' -o ',' -v 'price/@currency' -n library.xml

b1,29.50,USD
b2,44.00,USD
b3,250000,VND

Three books, three lines, a comma-separated extract with no script. This is the shell equivalent of the Python loop in converting XML to CSV for spreadsheet users, with one important difference: nothing quotes the output. The title SQL Tuning, 2nd Edition contains a comma, and if you printed it in this template, the line would have an extra field. For anything that goes to a CSV reader, use a tab as the separator (-o $'\t') and a tool that splits on tabs, or switch to a script that uses a CSV library.

The escaping surprise

By default xmlstarlet prints values the way an XML tool would, with special characters escaped, even in text mode:

xmlstarlet sel -t -v '//book[1]/note' -n library.xml        # Covers R&D
xmlstarlet sel -T -t -v '//book[1]/note' -n library.xml     # Covers R&D
xmllint --xpath 'string(//book[1]/note)' library.xml        # Covers R&D

Covers R&D
Covers R&D
Covers R&D

The first command printed the escaped form, &. Add -T (text output) before -t and it prints the plain text, matching xmllint's string(). If you pipe default output into another program, you will see & and < in your data, and a grep for R&D will find nothing. The fix is the one flag. (xmllint --xpath on a node-set, without string(), prints the XML of the node, with the escaping and the tags, which is another way to get surprised: xmllint --xpath '//book[1]/note' printed <note>Covers R&amp;D</note>.)

Missing values

A book without a note, and one with an empty note, both print as an empty line when you ask for the value:

xmlstarlet sel -t -m '//book' -v 'note' -n library.xml | cat -A

Covers R&D$
$
$

(cat -A shows line ends as $.) The second book has no note element and the third has an empty one, and the output cannot tell them apart. If the difference matters, select with predicates in separate runs: //book[note] printed b1 and b3, and //book[not(note)] printed b2, so the empty note counts as present.

Counting and listing paths

For a number, ask XPath for it:

xmlstarlet sel -t -v 'count(//invoice[@cur="USD"])' -n invoices.xml

12138

On the 4.4 MB invoices file that took 0.12 seconds, and on the 36.6 MB orders file count(//order) returned 200000 in 1.1 seconds. The tool builds a tree, so memory grows with the file (the earlier measurements put it at 15 to 20 times the size), and for files of several hundred megabytes a streaming script is the better choice.

A feature I use constantly is el, which lists the elements of a document:

xmlstarlet el -u invoices.xml

invoices
invoices/invoice
invoices/invoice/customer
invoices/invoice/due
invoices/invoice/line
invoices/invoice/line/amount
invoices/invoice/memo

The -u flag lists each distinct path once. Seven lines tell you the whole structure of a 4.4 MB file. Add -a to include the attributes (invoices/invoice/@id, invoices/invoice/@cur) and -v to include sample values. It is a profile without counts; for the counts, see counting elements and attributes in an unknown XML file.

Sorting and filtering

Sorting is part of the template. Sort the first three invoices by currency:

xmlstarlet sel -t -m '//invoice[position()<=3]' -s A:T:- '@cur' -v '@id' -o ' ' -v '@cur' -n invoices.xml

1 EUR
3 EUR
2 USD

-s A:T:- '@cur' sorts ascending (A), as text (T), with no case rule (-), on the currency. The sort applies to the matches of the enclosing -m. The filtering is all XPath, so everything in XPath predicates for position and text works: -m '//invoice[@cur="VND" and count(line) > 2]' selects what you would expect.

Namespaces

A default namespace has no prefix, so a path has to be given one with -N: xmlstarlet sel -N s=http://www.sitemaps.org/schemas/sitemap/0.9 -t -m '//s:url' -v 's:loc' -n sitemap.xml printed the three URLs of the sample sitemap, and the unprefixed //url printed nothing (wc -l gave 0). It is the same trap as everywhere else (how XML namespaces work). Note that the third URL came out as https://example.test/library/b?x=1&y=2, the escaped form again, so add -T here too.

Using it in scripts

Two habits keep shell use of the tool safe. Quote every XPath in single quotes, so the shell does not expand $ or split on spaces. And never splice a value you received (a file name, a form field) into the XPath string: //book[@id="$1"] built by interpolation is XPath injection, the same family as the injection described in sanitizing data you put into an XML document, and a value containing a quote ends the string early. Validate the value against an allow list first, or switch to a script that passes it as a parameter. Also check the exit status: xmlstarlet returns non-zero for a parse error, and set -e or an if makes a pipeline stop instead of continuing with an empty variable.

Related reading

For editing with the same tool see editing XML files in place from a script, and for the toolbox handy command-line tools for XML. The tool's reference is the xmlstarlet user's guide, and the XPath it evaluates is defined in the XPath 1.0 specification. Test the XPath in the XPath tester.

When xmlstarlet is the wrong tool for the extraction

The output must be valid CSV or JSON. The template prints text with no quoting. Use a script with a CSV or JSON library.

The file is huge. It parses the whole tree. Use a streaming parser.

You need grouping or arithmetic beyond XPath 1.0. sum and count are there; group by is not. Use a script or load the data into a database.

The tool is not installed. It is a package on most Linux distributions and in Homebrew, and absent from minimal containers. xmllint is more likely to be there, and lxml more likely still.