Counting Nodes in XML From the Shell Without Miscounting

"How many records are in this file?" is the first question about any XML export, and the shell tempts you with grep -c. It prints a number, the number is plausible, and it is often wrong. Counting is a place where the difference between text tools and XML tools matters, and the wrong answer is silent.

Why grep -c is wrong

grep -c pattern counts the lines that contain the pattern, not the occurrences. XML does not respect lines. An exporter may write a whole record on one line, or a whole file on one line, or split one element across five.

grep -c '<line' invoices.xml
grep -o '<line' invoices.xml | wc -l
xmllint --xpath 'count(//line)' invoices.xml

20000
49819
49819

The first command printed 20000: the file has one invoice per physical line, and each invoice line holds one to four line elements, so grep -c counted invoices. The second counts every match and returns 49,819, and the third, an XPath count over the parsed tree, agrees. On the extreme case, a file that is all on one line:

printf '<r><i/><i/><i/></r>' > one.xml
grep -c '<i' one.xml          # 1
grep -o '<i' one.xml | wc -l  # 3
xmllint --xpath 'count(//i)' one.xml   # 3

1
3
3

grep -c says there is one i in a document that has three. If the XML is pretty-printed with one element per line, grep -c happens to be right, which is how the bad habit survives: it works on the test file you formatted by hand.

Why grep -o is not enough either

grep -o '<line' | wc -l fixed the line problem and has others. The pattern <line also matches elements whose names start with it, such as <lineTotal> or <lines>. It matches inside comments (<!-- <line> removed -->) and inside CDATA sections, where the text only looks like an element. It matches elements in any namespace, and it counts both <line> and <line/>, which is what you want. A literal <line in text is written with an entity, so it does not match, which is correct. A pattern with the delimiter, '<line[ />]', removes the prefix problem (it matches <line and <line> and <line/), and the comment and CDATA cases remain. For a quick estimate, grep -o '<line[ />]' | wc -l is acceptable. For a number you will quote, use a parser.

Counting with a parser

Three tools, one answer:

xmllint --xpath 'count(//line)' invoices.xml
xmlstarlet sel -t -v 'count(//line)' -n invoices.xml
python3 -c "from lxml import etree; print(etree.parse('invoices.xml').xpath('count(//line)'))"

All print 49819 (the Python one as 49819.0, since XPath numbers are floats). XPath counts what you ask, with the structure available. Counts with conditions are one expression away:

xmllint --xpath 'count(//invoice[@cur="USD"]/line)' invoices.xml          # 30245
xmlstarlet sel -t -v 'count(//invoice[@cur="USD"])' -n invoices.xml        # 12138

30245
12138

30,245 US dollar lines in 12,138 US dollar invoices. These do exactly what the XPath says, so they are the numbers to trust and to cross-check against a streaming count.

A reminder about namespaces: count(//url) on a sitemap returns 0 because the elements are namespaced, and grep -o '<url>' | wc -l returns the right number, which makes grep look better than the parser. The fix is the namespace-aware path, or local-name(): count(//*[local-name()="url"]). A zero from xmllint on a file you know is not empty is a namespace until proven otherwise.

How fast and how big

Parsing is cheap, and the cost is in memory, not time.

time xmlstarlet sel -t -v 'count(//invoice[@cur="USD"])' -n invoices.xml     # 0.12 s
time xmlstarlet sel -t -v 'count(//order)' -n /tmp/orders_nested.xml          # 1.1 s, 200000
time xmllint --stream --noout invoices.xml                                    # 0.03 s

The 36.6 MB file with 200,000 orders counted in 1.1 seconds, and the 4.4 MB file in 0.12 seconds. The last line is a well-formedness check in stream mode, which does not build a tree and finished in 0.03 seconds, the cost of reading the file. The tree-based counters use roughly 15 to 20 times the file size in memory, which is fine at these sizes and not fine at 2 GB.

Counting without a tree

For files too big for a tree, count in a stream. The same idea gives a count per element name, which is also the first profile of an unknown file:

import collections
from lxml import etree
n = collections.Counter()
for _, el in etree.iterparse("invoices.xml", events=("end",)):
    n[el.tag] += 1
    el.clear()
print(n.most_common())

[('amount', 49819), ('line', 49819), ('customer', 20000), ('due', 20000), ('invoice', 20000), ('memo', 400), ('invoices', 1)]

One pass, flat memory, and every element name counted. It reports 49,819 for both line and amount, 20,000 for invoice, customer and due, and 400 for memo, the same counts as the longer profile in counting elements and attributes in an unknown XML file. xmlstarlet el -u lists the distinct paths without counts, which is a fast way to see the shape first (extracting values with xmlstarlet).

Checks worth making

A count is a claim about the file, and a few comparisons make it solid.

• Agree with a second method. An XPath count and a streaming count should match. If grep -o is close but not equal, find the difference: it is usually a comment or a prefix.

• Compare with the sender's number. Exports often say how many records they contain, in a header, a filename or an email. A mismatch is the most important finding of the day.

• Count before and after any step. A conversion, a load or a filter changes the count, and the change should be explained: 49,819 in, 49,819 rows out, or a list of rejects.

• Count the distinct, not just the total. count(//invoice/@id) and the number of distinct ids differ if the file has duplicates. In XPath 1.0, count distinct with a streaming script or load the ids.

Related reading

For the broader toolbox, see handy command-line tools for XML. The counting recipes feed the analysis methods in analysis on XML data when nobody has converted it yet. XPath's count and the other core functions are in the XPath 1.0 function library, and grep's counting behavior in the GNU grep manual. Test an expression in the XPath tester.

When a count should come from the parser, not the shell

You are going to quote the number. If it appears in a report, an email or a reconciliation, count with a parser and cross-check.

The file may contain comments, CDATA or lookalike names. Text tools cannot tell markup from content that looks like markup.

There are namespaces. Parsers need a namespace map and grep does not, which means grep can be right when the parser reports zero. Fix the query, not the tool.

A quick look is enough. For "is this about a thousand or a million", grep -o | wc -l is fine. Use the cheapest tool that is accurate enough, and know which it is.