Reading XML in Ruby With Nokogiri and Strict Parsing

Ruby's XML story is Nokogiri. It wraps libxml2, offers CSS selectors and XPath, handles HTML as well as XML and is installed in most Rails applications. It is convenient and has two defaults worth overriding when you read data.

Strict parsing and the read

require "nokogiri"
require "bigdecimal"
doc = Nokogiri::XML(File.read("../library.xml")) { |c| c.strict.nonet }
usd = BigDecimal("0")
doc.xpath("//book").each do |b|
  note = b.at_xpath("note")
  puts [b["id"], b.at_xpath("title").text, b.xpath("author").map(&:text).join("; "),
        "#{b.at_xpath('price').text} #{b.at_xpath('price')['currency']}",
        "available=#{b['available'] == 'true'}",
        "note=#{note.nil? ? '<absent>' : "'#{note.text}'"}"].join(" | ")
  usd += BigDecimal(b.at_xpath("price").text) if b.at_xpath("price")["currency"] == "USD"
end
puts "USD total #{usd.to_s('F')}"

b1 | Cache Patterns | Ana Lee; Bo Chen | 29.50 USD | available=true | note='Covers R&D'
b2 | SQL Tuning, 2nd Edition | Cy Ortiz | 44.00 USD | available=false | note=<absent>
b3 | Indexes "Explained" | Di Park | 250000 VND | available=true | note=''
USD total 73.5

The block passed to Nokogiri::XML sets parse options. strict makes errors fatal (the next section shows why), and nonet forbids network access. xpath returns a node set and at_xpath returns the first node or nil, which is the right tool for an optional element: note.nil? separates the absent note (book 2) from the empty one (book 3, whose node exists with empty text). Attributes read like hash keys, b["id"]. Money goes through BigDecimal, so 29.50 plus 44.00 is exactly 73.5; a Float would work for this pair and drift on longer sums. to_s('F') prints it in plain decimal notation, since BigDecimal#to_s defaults to scientific.

The authors were collected with xpath("author"), so a book with one author is the same code as one with two. As with every API, the text of the amount is a string until you convert it.

Lenient by default

This is the default most people do not know about. Nokogiri's XML parser is built to recover: it repairs broken documents and hands you a tree. I parsed a document with a mismatched tag both ways:

begin
  Nokogiri::XML("<a><b></a>") { |c| c.strict }
rescue Nokogiri::XML::SyntaxError => e
  puts "strict -> #{e.message.lines.first.strip}"
end
lenient = Nokogiri::XML("<a><b>x</a>")
puts "lenient -> #{lenient.to_xml.gsub("\n", ' ')} | errors: #{lenient.errors.size}"

strict -> 1:11: FATAL: Opening and ending tag mismatch: b line 1 and a
lenient -> <?xml version="1.0"?> <a>   <b>x</b> </a>  | errors: 1

Strict mode raised Nokogiri::XML::SyntaxError with the position and the reason. The default mode returned a document in which the b element had been closed for me, with one entry in lenient.errors. Nothing was raised. If you process data from a partner, the lenient parse turns a truncated file into a shorter list of records that looks fine, which is the silent-loss mode every loader should avoid. Use strict for data and keep the lenient default for HTML and for tools that show what is parsable (fixing XML parse errors fast covers repair deliberately). If you must be lenient, check doc.errors.empty? and treat a non-empty list as a failure to report.

Namespaces

ns = Nokogiri::XML(File.read("../lib_ns.xml"))
puts "no ns: #{ns.xpath('//book').size} | with ns: #{ns.xpath('//l:book', 'l' => 'urn:example:library').size} | remove_namespaces!: #{ns.remove_namespaces!.xpath('//book').size}"

no ns: 0 | with ns: 2 | remove_namespaces!: 2

//book found nothing, because the elements are in urn:example:library. Passing a hash of prefix to URI as the second argument of xpath fixes it, and the prefix is yours. Nokogiri also offers remove_namespaces!, which strips all namespaces from the document so that plain paths work. It is a popular shortcut in scripts and it has a cost: if two vocabularies share local names (an Atom link and an HTML link), they now collide. Use it when you control the data and know there is one vocabulary, and the prefix map when you do not. For a document whose root namespace is the default one, doc.collect_namespaces returns the prefixes declared, which saves a trip to the file.

Entities

I ran the three-line external entity document through two configurations:

external entity (default): ""
external entity (noent): "9d9bda84ebcf\n"

By default the entity was not expanded and the value was empty. With noent, which tells libxml2 to substitute entities, the result was the container's host name, read from /etc/hostname. So the same warning as in PHP applies: the option whose name suggests "no entities" substitutes them, and it opens external file reads. Do not pass noent for untrusted input, and include nonet as above. If you see noent in a code base, find out why it was added; it is often a leftover from an XML sample that "wouldn't show the text" (what an XML external entity attack is).

Streaming with Reader

Nokogiri::XML builds a tree. For large files there is a pull-style reader:

n = 0
Nokogiri::XML::Reader(File.open("../orders_nested.xml")).each do |node|
  n += 1 if node.name == "order" && node.node_type == Nokogiri::XML::Reader::TYPE_ELEMENT
end

Reader: 200000 orders in 1.38s

200,000 orders in 1.38 seconds without building a tree. The reader yields every node, including end tags and text, so you filter on node_type, and you can call node.outer_xml on an element of interest to get its XML and parse that small fragment with Nokogiri::XML for XPath on the record. That is the Ruby version of the iterparse pattern. The SAX interface (Nokogiri::XML::SAX::Parser) is the push alternative.

Testing the parse

Two tests earn their place in any Ruby code that reads partner XML. The first loads a sample file with strict and asserts on the extracted values, including the absent-note and one-author cases. The second feeds the same code a truncated copy and asserts that it raises, so a change that quietly switches the parser to lenient mode is caught. Add the external entity file as a third test and assert that the text is empty. They take a few lines each and turn the parser's defaults into documented, checked behavior.

Practical notes

Encoding. Pass bytes and let the parser read the declaration (File.binread, or give Nokogiri::XML an IO). Reading with File.read in a locale that is not UTF-8 can decode text wrongly before the parser sees it.

Errors. doc.errors holds Nokogiri::XML::SyntaxError objects with line, column and message.

CSS selectors. doc.css("book > title") works on XML as well as HTML, and is fine without namespaces.

Gem and system libraries. The precompiled gem bundles its own libxml2, so behavior depends on the gem version (Nokogiri::VERSION_INFO shows the libxml version). Pin it.

Related reading

For the language overview see parsing XML in Java, C# and other typed languages, and for the same strictness question in Python safe XML parsing in Python with defusedxml. Nokogiri is documented at nokogiri.org and its parsing options in the ParseOptions reference. Test an XPath in the XPath tester.

When Nokogiri is more than a Ruby script needs

The format is trivial and you control it. Ruby's standard library includes REXML, which suits small configuration files; I did not test it here.

The file is huge. Use Reader or SAX, not Nokogiri::XML.

You want objects. A mapping library on top of Nokogiri (several exist) saves code, at the cost of another dependency; I did not test one.

You are parsing HTML. Use Nokogiri::HTML5, which follows the browser's rules, not the XML parser.