Running XSLT in Python With lxml and What It Reports

If you already use lxml to parse XML, XSLT is a short step away: the library wraps libxslt, and a stylesheet becomes a Python callable. This is the easiest way to embed a transformation in a script or a web application, and it has a few behaviors that differ from the command line.

Compile once, call many times

from lxml import etree
doc = etree.parse("library.xml")
xslt = etree.XSLT(etree.parse("titles.xsl"))
result = xslt(doc)
print(str(result))

etree.XSLT(stylesheet_tree) compiles the stylesheet and returns an object you call with a document. The compile step parses and checks the stylesheet and is the expensive part, so do it once at start-up and reuse the object for every document. The call returns a result tree (an _XSLTResultTree). What you do with it depends on the output method:

• str(result) gives the output as text: for method="text", the text; for XML, the serialized document including the declaration.

• bytes(result) gives the output encoded as the stylesheet's xsl:output encoding says, which is the right thing to write to a file or an HTTP response.

• result.getroot() gives the root element when the output is XML, so you can keep working on the tree with XPath.

The distinction between str and bytes matters for encoding. str(result) yields a Python string; if you write it to a file opened in text mode, the platform's encoding decides the bytes, and the XML declaration may disagree with them. Write bytes(result) to a file opened in binary mode (encoding mismatch errors in XML documents).

Parameters are expressions

The keyword arguments you pass to the call become stylesheet parameters, and each value is an XPath expression:

xslt = etree.XSLT(etree.parse("params.xsl"))
str(xslt(doc, min="50", cur=etree.XSLT.strparam("VND")))
str(xslt(doc, min="100000", cur="'VND'"))
str(xslt(doc, cur="VND"))

'min=50 type-test: 1\nbooks=3\n'
'min=100000 type-test: 1\nbooks=3\n'
'min=30 type-test: 0\nbooks=3\n'

The first two calls matched the one book in dong: one passes the string with strparam, the other quotes it by hand. The third passed cur="VND" bare, which XPath reads as a path to elements named VND, an empty node-set, and the book count was 0. Nothing raised. Use etree.XSLT.strparam(value) for every string, and especially for anything that came from a user, because a string containing a quote breaks an expression, and a crafted one can change what the expression means. The same trap on the command line, and the number-versus-string comparison rules, are in passing parameters into an XSLT stylesheet.

Messages and errors live in the log

xsl:message inside a stylesheet is the debugging tool of choice, and with xsltproc it prints to standard error. In lxml it does not print. It goes to the XSLT object's error_log:

d3 = etree.XSLT(etree.parse("dbg3.xsl"))
r = d3(doc)
print("message log:", [(e.type_name, e.message) for e in d3.error_log])

message log: [('ERR_OK', 'matched library with 3 books')]

The message text is there with a type name of ERR_OK, which is how libxslt labels a message that is not an error. A script that expects to see messages in the terminal will see nothing and conclude the stylesheet said nothing. After a call, read xslt.error_log; with terminate="yes" on the message, the call raises XSLTApplyError and the log has the text. For a malformed stylesheet, etree.XSLT(...) raises XSLTParseError, whose message and error_log describe the problem. Not everything is caught at compile time: in my test a call to an unknown function (nofunc(.)) did not fail when the stylesheet was compiled, so run each stylesheet once on a sample at start-up. Clear the habit of passing a stylesheet through etree.XSLT inside a request handler; compile at import time so a bad stylesheet fails at deploy.

Access control

A stylesheet can read other files with document() and, depending on settings, fetch URLs. If the stylesheet is yours, that is a feature. If it comes from a user or a partner, it is a way to read your files. lxml lets you forbid it:

ac = etree.XSLTAccessControl(read_network=False, read_file=False)
sty = etree.XSLT(stylesheet_with_document_call, access_control=ac)
sty(doc)

apply error: xsltLoadDocument: read rights for file:///w/library.xml denied

With read_file=False the document('library.xml') call failed with an XSLTApplyError that names the file and says rights were denied. The control object also has write_file, create_dir and write_network, all of which cover the EXSLT write extensions. Combine it with the safe parser settings for the input documents (disabling DTDs and external entities in Java and Python); the access control governs what the stylesheet does, and the parser options govern what the document does. For an untrusted stylesheet the better answer is not to run it at all, since XSLT is a programming language and libxslt has had its share of vulnerabilities.

Speed: where lxml's XSLT stops being the right tool

The grouping stylesheet from grouping records with XSLT 2 and Muenchian keys, applied through lxml to the 4.4 MB file of 20,000 invoices:

group10 via lxml on 4.4 MB: 4.26s EUR: 3897 invoices, 9764 lines
python loop on same tree: 0.031s {'EUR': 3897, 'USD': 12138, 'VND': 3965} {'EUR': 9764, 'USD': 30245, 'VND': 9810}

Same answer, 4.26 seconds against 0.031 for a Python loop over the already parsed tree. The lxml transform is a thin wrapper around xsltproc's engine, and the engine is slow on this idiom. That is not an argument against XSLT in Python; for restructuring documents the speed is fine. It is an argument against doing aggregation in 1.0 stylesheets when you have Python in hand. A division of labor that works: XSLT for the structural changes that are awkward in code (wrapping, renaming, moving subtrees, producing HTML), Python for counting, grouping and arithmetic.

You can also call Python from the stylesheet with extension functions (etree.FunctionNamespace), which is useful for things XSLT 1.0 lacks, such as a regular expression test or a date parser. I did not run an extension here. They are a way to extend the engine, and they also make the stylesheet dependent on your Python code, so the stylesheet no longer runs under xsltproc.

A small wrapper

A function that bakes in the habits above:

def transform(xslt, doc, **params):
    safe = {k: etree.XSLT.strparam(v) if isinstance(v, str) else v for k, v in params.items()}
    result = xslt(doc, **safe)
    for e in xslt.error_log:
        log.info("xslt: %s", e.message)
    return result

Strings are wrapped in strparam, other values are passed as they are (a number becomes an expression only if you convert it to a string such as "50", so pass numbers as strings and strings as strparam; adjust the wrapper to your convention). The log is read after every call.

Related reading

For the command-line equivalent see running XSLT from the command line, and for the cluster overview XSLT and XQuery for real transformations. The XSLT interface is documented in the lxml XPath and XSLT page, and the engine in the libxslt project. For XPath without a stylesheet, see XPath in Python with lxml. Test an XPath in the XPath tester.

When lxml's XSLT 1.0 is not enough

You need XSLT 2.0 or 3.0. Grouping with for-each-group, regular expressions, format-date, maps and streaming need Saxon (a Java process, or SaxonC bindings). From Python, call it as a subprocess or use the vendor's Python package.

You need speed on large documents. The tree is built in memory, and the engine's cost grows with the stylesheet's idioms. Stream the data in Python and use XSLT only for fragments.

You do not control the stylesheet. Do not run it. If you must, use access control, a size limit, a time limit in a separate process, and no network.

The job is a data conversion. A mapping written in Python is easier to test and to debug than a stylesheet.