Editing XML Files in Place From a Script Without Damage

Scripts edit XML files all the time: bumping a version in a build file, switching a setting in a config, adding a dependency, changing a URL in a feed. The tempting tool is sed, because it is there and the edit is a one-liner. It works until the value contains an ampersand, or the pattern matches in a comment, or the same text appears in two places. A tool that parses the XML first avoids all of that, and a few of them are on every machine that has xmllint.

A structural edit with xmlstarlet

xmlstarlet ed applies edits and prints the result. The actions are -u (update), -i/-a (insert before or after), -s (subnode), -d (delete), -r (rename), each taking an XPath. Without -L the original is not touched:

xmlstarlet ed -u '//book[@id="b2"]/price' -v '39.00' library.xml | grep -n 'b2' -A4

10:  <book id="b2" available="false">
11-    <title>SQL Tuning, 2nd Edition</title>
12-    <author>Cy Ortiz</author>
13-    <price currency="USD">39.00</price>
14-  </book>

The new price is in the printed copy. This is the safe way to preview an edit: look at the output, then add -L to write it. With three edits at once and -L on a copy:

cp library.xml lib_edit.xml
xmlstarlet ed -L -u '//book[@id="b2"]/price' -v '39.00' \
  -i '//book[@id="b3"]' -t attr -n 'featured' -v 'yes' \
  -d '//note[not(node())]' lib_edit.xml
diff library.xml lib_edit.xml

13c13
<     <price currency="USD">44.00</price>
---
>     <price currency="USD">39.00</price>
15c15
<   <book id="b3" available="true">
---
>   <book id="b3" available="true" featured="yes">
19d18
<     <note/>

Three hunks for three edits: the price changed, the attribute was added, the empty note was deleted. The XPath selected each target by what it is (@id="b2", not(node()) for an element with no content), so a reordering of the file would not break it. The diff is clean here because the original was already formatted the way xmlstarlet writes it (two-space indent). A file formatted differently comes back reindented, and the diff then shows the layout change on every line along with the edit. Format the file once, commit that, and then let scripted edits produce small diffs (formatting XML with xmllint for readable diffs).

What sed does instead

The same price change with sed:

sed 's#<price currency="USD">44.00#<price currency="USD">39.00#' library.xml | diff library.xml -

13c13
<     <price currency="USD">44.00</price>
---
>     <price currency="USD">39.00</price>

On this file the result is identical to the tool's, and it is the situation where sed looks fine. The pattern worked because the text is on one line and appears once. Then the differences start. If the price were 44.00 in two books, sed changes both, and the XPath version changed the one you named. If the element were split over lines, or an attribute reordered, the pattern would miss it without any message. And the replacement text is inserted as typed. Updating a value through xmlstarlet with the string a & b < c:

xmlstarlet ed -u '//v' -v 'a & b < c' cm.xml

<?xml version="1.0"?>
<c>
  <!-- keep me -->
  <v>a &amp; b &lt; c</v>
</c>

The tool wrote the escapes for me. sed would have written a & b < c as is, and the file would no longer be well-formed (and in sed's replacement part, an unescaped & stands for the whole matched text, so it would also have inserted the pattern in place of the ampersand). Notice also that the comment keep me survived, because the tool works on the full tree including comments. That comes with a price: the file is rewritten from the tree, so whitespace and the exact form of the declaration, empty elements and entity references may change, which is where unrelated lines show up in the diff.

The rule I follow: use sed for text files and for XML only when the change is a literal string that cannot appear anywhere else and cannot contain markup characters. Everything else goes through a parser.

The same in Python

When the edit has logic, use lxml:

from lxml import etree
tree = etree.parse("library.xml")
for price in tree.xpath('//book[@id="b2"]/price'):
    price.text = "39.00"
tree.xpath('//book[@id="b3"]')[0].set("featured", "yes")
for note in tree.xpath("//note[not(node())]"):
    note.getparent().remove(note)
tree.write("lib_edit.xml", encoding="UTF-8", xml_declaration=True)

lxml keeps comments, processing instructions and the namespace prefixes as written, and does not reindent unless you pass pretty_print=True, so the diff of an edit stays small if the input already had its whitespace. A caveat for removal: remove takes the element's tail text with it. On <r>\n <a/>after-a\n <b/>\n</r>, removing a left <r>\n <b/>\n</r>, and the words after-a were gone. In data files the tail is only whitespace and the loss is harmless; in mixed content it deletes real text, so move the tail to the previous sibling or the parent before you remove. Run xmllint --noout on the result.

Doing it safely

An edit script is a small program with the power to damage a file, so some habits earn their keep.

Write a new file and rename. xmlstarlet ed -L edits in place, which is fine for a copy under version control and risky for the only copy of a production file. Write file.xml.new, validate it, then mv it over the original, which is atomic on the same filesystem.

Validate after the edit. At least xmllint --noout, and against the schema if there is one (validating XML against XSD with xmllint). An edit that produced a valid file with the wrong content is the hard case, which is why you also check the diff.

Check that the edit matched something. xmlstarlet ed prints the document and exits 0 even when the XPath selected nothing: the file comes back unchanged. For a script that must change something, count the matches first (xmlstarlet sel -t -v 'count(//book[@id="b2"])') and fail on zero. Silent no-ops are the characteristic bug of scripted edits.

Keep the namespace in mind. An XPath without the namespace prefix on a namespaced file selects nothing, and the edit silently does not happen. Use -N with xmlstarlet or a namespace map in lxml.

Keep a copy and a diff. A script that edits many files should log the file list and keep a backup directory, or, better, run inside a version-controlled tree, so git diff is the review.

Related reading

The toolbox is surveyed in handy command-line tools for XML, and the reading side in extracting values with xmlstarlet. For editing the document model in a program, see building an XML document in Python from data. The lxml tree API is described in its tutorial, and xmlstarlet's edit command in the user's guide. Try the XPath in the XPath tester.

When a text edit is the better choice

The file is XML in name only. Templates that mix placeholders into XML ({{ value }}, ${var}) are not well-formed until rendered. A parser cannot load them, so edit the text.

You must preserve every byte. A parse and write changes whitespace, entity forms and the declaration. If the exact bytes matter (a signed file, a vendored file you diff against upstream), use a text edit with a pattern you have proven unique, and check the diff.

The change is a one-off by hand. For a single edit in an editor, the editor's find and replace is fine. Structural tools are for edits that repeat.

The edit changes the schema's meaning. Renaming an element or changing a type is a migration, not an edit. Version it and test the consumers (testing feed compatibility before a schema change).