Using yq to Read XML From the Command Line and Pipelines

yq is a command-line processor for structured data, and the name is shared by two unrelated programs that read XML. One is a Python wrapper around jq by a different author, installed with pip, which comes with a command called xq for XML. The other is a single Go binary, mikefarah/yq, available as a package or a container image, which reads XML directly. A command copied from an answer that assumed one will fail, or worse, give different keys, on the other. This article shows the output of each on the same file so that you can tell which one you have and what to expect.

Which yq do you have

yq --version

The Go tool prints yq (https://github.com/mikefarah/yq/) version v4.54.1. The Python one prints yq 4.4.5 followed by the version of the jq binary it wraps (jq-1.6 here). The Python one is a thin layer: yq runs jq on YAML, and xq runs jq on XML, both converting the input to JSON and handing it to the jq binary. The Go one has its own expression language, modeled on jq's but not identical, and reads many formats with one binary. If you cannot remember which you installed, yq --version tells you.

The Python tool: xq

xq '.library.book[0]' library.xml

{
  "@id": "b1",
  "@available": "true",
  "title": "Cache Patterns",
  "author": [
    "Ana Lee",
    "Bo Chen"
  ],
  "price": {
    "@currency": "USD",
    "#text": "29.50"
  },
  "note": "Covers R&D"
}

xq converts the XML to JSON with the convention that attributes get an @ prefix and the text of an element that also has attributes goes under #text, the same as the xmltodict library produces. Then the expression is plain jq. Because the keys begin with @ and #, you must quote them in filters: .library.book[0]."@id". To get YAML output, add -y: xq -y '.library.book[0]' library.xml printed the same structure as YAML with '@id': b1 and so on. The shape rules are those of the JSON conversion article: an element that occurs twice becomes an array, and an element that occurs once is a plain value, so .author[] works on the first book and fails on a book with one author.

The Go tool

yq -p=xml -o=json '.library.book[0]' library.xml

{
  "+@id": "b1",
  "+@available": "true",
  "title": "Cache Patterns",
  "author": [
    "Ana Lee",
    "Bo Chen"
  ],
  "price": {
    "+content": "29.50",
    "+@currency": "USD"
  },
  "note": "Covers R&D"
}

The structure is the same and the key names are not: attributes are +@id (a plus sign, then an at sign), and the element text beside attributes is +content. These prefixes are configurable (--xml-attribute-prefix and --xml-content-name) but the defaults differ from the Python tool's, so a filter written for one needs different keys on the other. Selecting values works as in YAML:

yq -p=xml '.library.book[] | .title' library.xml

Cache Patterns
SQL Tuning, 2nd Edition
Indexes "Explained"

This printed the three titles, one per line, in the output format the tool chose. It also printed a warning on standard error about its default output, which in this version says that the default is now auto-detected from the file name and that for backward compatibility yaml was set; name the output format to silence it and to make the command independent of that default (-o=yaml, -o=json, or -o=xml if you want XML back).

Empty and absent

The two tools keep the difference between a missing element and an empty one, in different ways. In the Go tool, selecting note on the third book (which has <note/>) printed an empty line, and on the second book (which has no note) printed null:

null

So the empty element is an empty string and the absent one is null, and a script that tests for "has a note" has to decide which of the two counts. The Python tool gives null for an empty element and no key for an absent one, the same distinction written differently. Whichever tool you use, write the test for both.

Lists of one

Both tools build arrays from repeated elements, so the shape depends on the data:

yq -p=xml -o=json '.library.book[].author' library.xml

The first book gives an array of two names and the others give a string each, so a later jq stage that expects arrays has to normalize first. In jq the idiom is (.author | if type == "array" then . else [.] end); in the Go tool, .author | ... with tag checks. The cleaner approach is to avoid depending on the shape and select by path from the XML, which is what XPath tools do. If you need XPath, xmllint and xmlstarlet are the better tools (extracting values with xmlstarlet).

What yq does well

The strength of this approach is the pipeline you already know. You can filter, map and reshape with jq expressions, convert to CSV with @csv, and combine XML with JSON and YAML data in one tool. For reading a config file, listing a handful of values from a feed or comparing a field across a directory of files, it is a short command. For formats the Go tool supports natively, it converts without leaving the shell: XML to YAML for a review (converting XML to YAML for config review), XML to JSON for a script.

Where it stops

The conversion is lossy in the same ways as every XML-to-JSON mapping: order between different element names, comments, namespaces and the difference between attribute and element are encoded by convention, and mixed content does not survive. Namespaces keep their prefixes in the key names ("ns:title"), so a file whose producer changes the prefix changes your keys. Everything is a string; numbers need an explicit conversion. And both tools read the whole file into memory.

Related reading

The family of tools is surveyed in handy command-line tools for XML. The JSON conventions behind these outputs are in converting XML to JSON and the attribute problem. The Python project is described at the yq repository by kislyuk, and the Go tool in the mikefarah/yq documentation on XML. Test the XPath you would use instead in the XPath tester.

When yq is the wrong way to read XML

The XML has namespaces you must tell apart. The prefix-in-key convention breaks when two vocabularies share local names or when prefixes change. Use an XPath tool with a namespace map.

The file is document-like. Mixed content and element order do not map to keys. Use XSLT or a proper parser.

The data is large. Both tools load the whole file. Stream with a script.

You need exact types. Convert explicitly, or read with a typed parser.