Converting JSON to XML With Sensible Element Names

JSON and XML look similar enough that the conversion seems trivial: an object becomes an element, its keys become children. Then a key arrives with a space in it and the converter fails, or an array with one item loses its list-ness, or a null turns into nothing. This article lists the decisions in order and shows what each tool did with them.

Start with the document

doc = {"order": {"id": 7, "customer name": "Ana Lee", "2fa": True, "tags": ["a", "b"],
                 "note": None, "lines": [{"sku": "A-1", "qty": 2}], "total": 12.5}}

Everything that goes wrong is in the keys and values of this one object. Ask the standard converter first:

import xmltodict
print(xmltodict.unparse(doc, pretty=True))

unparse error: ValueError Invalid element name: whitespace not allowed

It refused, which is the better outcome. A converter that wrote <customer name>Ana Lee</customer name> would have produced a file no parser can read.

Decision 1: what is a legal name

An XML element name must start with a letter, an underscore or (in the full specification) certain other Unicode letters, and may continue with letters, digits, hyphens, underscores and periods. It cannot contain spaces, and a colon is reserved for namespace prefixes. A name cannot start with a digit, and names beginning with xml in any case are reserved. lxml enforces this:

'a b' -> Invalid tag name 'a b'
'1x'  -> Invalid tag name '1x'
'x:y' -> Invalid tag name 'x:y'
'<>'  -> Invalid tag name '<>'

JSON keys have no such rules, so they arrive as customer name, 2fa, first.name, $ref, @type, an empty string or an emoji. The converter must either refuse, rename, or carry the original key in a different way. I write a small function and apply it everywhere:

import re
def name(k):
    k = re.sub(r"[^A-Za-z0-9_.-]", "_", k.strip())
    return k if re.match(r"[A-Za-z_]", k) else "_" + k

It replaces every illegal character with an underscore and puts an underscore in front of a name that starts with a digit or a hyphen. So customer name becomes customer_name and 2fa becomes _2fa. The function is deterministic, but it is not injective: customer name and customer_name both map to the same element, and two keys in one object can collide. For machine-generated JSON with known keys that is fine. For arbitrary JSON, detect the collision and either fail or add a suffix.

If you need to get the original key back, write it as an attribute: <field name="customer name">Ana Lee</field>. This "generic element" style is verbose and fully lossless, and many exchange formats use it. It is the right choice when the keys are data, for example a dictionary of arbitrary user-defined fields, and the wrong one when the keys are a fixed vocabulary, because then you want <customer_name> for readers.

Decision 2: arrays

An array has no counterpart in XML. The convention is to repeat the element: "tags": ["a", "b"] becomes two <tags> children. The conversion in my script:

def build(parent, key, val):
    if isinstance(val, list):
        for v in val:
            build(parent, key, v)
        return
    el = etree.SubElement(parent, name(key))
    ...

  <tags>a</tags>
  <tags>b</tags>

Two consequences. The element is named for the item, not the list, so tags appears twice and there is no wrapper, unless you add one (<tags><tag>a</tag><tag>b</tag></tags>), which is a style choice to make once. And an array of one item produces exactly the same XML as a plain value:

<r><i>only</i></r> <r><i>only</i></r>

Those are {"i": ["only"]} and {"i": "only"} through xmltodict.unparse, and they are identical. An empty array produces nothing. So the conversion cannot be reversed without extra information about which elements are lists, the same ambiguity as in converting XML to JSON and the attribute problem. If the receiver needs to know that tags is a list, put it in the schema or use the wrapper form.

Decision 3: null, booleans, numbers

JSON null is neither an empty string nor a missing value. In XML the options are to omit the element, write it empty, or mark it. Omitting loses the fact that the field was present. An empty element reads back as an empty string, which is a different value. The standard marker is xsi:nil="true" in the XML Schema instance namespace, and a simple version writes a plain attribute. My script wrote:

  <note nil="true"/>

With a namespace-declared xsi:nil it would be understood by schema-aware tools; without it, the attribute is a private convention the reader has to know about. For booleans, write the text true or false (XML Schema's lexical form, and what most readers accept). Numbers go out as their text, with str(12.5); check the float formatting for large and small values, since 1e-07 is valid for XML Schema double but not for a decimal. The XML has no type information after this, and a reader that cares needs the schema.

What came out

Here is the complete output of the script on the test document:

<order>
  <id>7</id>
  <customer_name>Ana Lee</customer_name>
  <_2fa>true</_2fa>
  <tags>a</tags>
  <tags>b</tags>
  <note nil="true"/>
  <lines>
    <sku>A-1</sku>
    <qty>2</qty>
  </lines>
  <total>12.5</total>
</order>

Read it as a reviewer. The names are legal. The array is two siblings. The object in lines became a lines element with the keys as children, and, because the list had one element, there is no way to tell from the XML that lines was an array. The null is explicit. The booleans and numbers are text. This is a usable document, and you can see what the receiver must be told: which elements repeat, which are numbers, and what nil means.

Escaping is the library's job

Text values are escaped by the library. A string a < b & c becomes a < b & c, and that is the reason to build XML with a library and not by joining strings (converting CSV to XML with a template shows the failure from the other direction). Characters that are not allowed in XML at all, such as a bell character or a NUL, make lxml raise All strings must be XML compatible. JSON allows them, escaped, in strings. Decide whether to strip them or reject the document; do not let them through, because no parser will accept the file.

Key order is preserved by Python's dictionaries, and XML consumers often rely on order, so keep the order of the source unless a schema says otherwise (an XSD sequence demands a specific order).

Naming conventions that read well

Choose a style and keep it. JSON keys are usually camelCase or snake_case, and XML vocabularies tend to use lowercase with hyphens or camelCase; mixing them looks careless. Pluralization needs a rule too: <lines><line>...</line></lines> reads better than <lines> repeated, and the wrapper gives you a place to put an attribute like count. Such rules live in code (a list of keys and the child name to use for each), not in a generic converter. A converter with options is usually less work than a dozen special cases, but only if the options cover your cases.

Related reading

For the opposite direction see converting XML to JSON and the attribute problem, and for the JSON-first view of the same conversions, converting JSON to XML. The naming rules are defined in the XML 1.0 specification, section 2.3, and the library used here is documented in the xmltodict repository. xsi:nil is defined in XML Schema Part 1. To check what a generated document selects, use the XPath tester.

When the XML has to follow a schema

A partner publishes an XSD. Then the element names, order, types and optionality are fixed. Write the mapping by hand against the schema and validate the output (validating XML against an XSD schema). A generic converter will produce a document that is close and invalid.

Namespaces are required. The converter above produces elements in no namespace. A schema with a target namespace needs the elements created in it, which is a different call in lxml (names in Clark notation).

Order matters. If the schema uses a sequence, emit children in that order, not in the order of the JSON keys.

Nothing downstream needs XML. If the only reason for the conversion is that one tool asked for it, check whether it also reads JSON before you take on the mapping.