Moving From XML to JSON Gracefully Without Losing Meaning

Teams move from XML to JSON for good reasons: a web client, a document store, a simpler toolchain. The move is usually done with a generic converter and a hope. Then the JSON is consumed by code, and code assumes one shape per field. The first record that arrives in a slightly different shape is a bug report. A conversion is a design exercise with a handful of decisions, and making them on purpose takes an afternoon.

What a generic converter decides for you

Take one order with two items and an empty note, and convert with xmltodict:

import json, xmltodict
x = '<order id="7"><item sku="A" qty="2">Pen</item><item sku="B">Ink</item><note/></order>'
print(json.dumps(xmltodict.parse(x), indent=1))

{
 "order": {
  "@id": "7",
  "item": [
   {
    "@sku": "A",
    "@qty": "2",
    "#text": "Pen"
   },
   {
    "@sku": "B",
    "#text": "Ink"
   }
  ],
  "note": null
 }
}

The conventions are visible. Attributes get an @ prefix, text next to attributes goes under #text, the empty note is null, and item is a list because it occurs twice. Each is a decision, and each can change with the data. An item with no attributes would be a string instead of an object with #text. A single item would not be a list at all (modeling lists and optional data in XML shows that trap). The @ and #text keys are legal JSON but awkward to use: order["@id"] works in a script, and many JSON tools and languages treat keys beginning with @ or # as special.

None of this is wrong. It is a mapping designed to be reversible, so XML could be rebuilt from the JSON. If your goal is JSON that people and code like to read, you want a different mapping.

Decide the rules first

Six questions settle the mapping. Write the answers down.

1. How are attributes named? Plain names (id), prefixed (@id, _id) or nested (attributes: {...}). Plain names are the friendliest, and they collide when an element has an attribute and a child of the same name. Pick the convention that collision is rarest in.

2. Which elements are lists? Name them. Everything else is a single value. This is the most important rule, and it should come from the schema or the documentation, not from the data in front of you.

3. What does an empty element become? null, an empty string, or the key omitted. XML has several states, absent, empty and nil, and JSON has null and absent (modeling lists and optional data in XML).

4. What happens to an element that has both attributes and text? Common choices: a value or text key, or the same #text.

5. How are numbers and booleans typed? XML text is text. Convert qty="2" to the number 2 only where the schema says integer, never by guessing, because a code such as 007 or a zip code 01234 must stay a string.

6. What happens to mixed content and namespaces? For mixed content, store the inner markup as a string (mixed content in XML and why it hurts mapping). For namespaces, drop prefixes and use local names unless two namespaces clash.

A small explicit mapping

With the rules written, the converter is short. This one uses plain attribute names, value for text next to attributes, null for empty elements, and a declared set of list element names that always become arrays under the plural name:

import json
from lxml import etree

LISTS = {"item"}

def to_obj(el):
    out = dict(el.attrib)
    kids = list(el)
    if not kids:
        text = (el.text or "").strip()
        if not out:
            return text or None
        if text:
            out["value"] = text
        return out
    for kid in kids:
        name = etree.QName(kid).localname
        val = to_obj(kid)
        if name in LISTS:
            out.setdefault(name + "s", []).append(val)
        else:
            out[name] = val
    return out

x = etree.fromstring('<order id="7"><customer>Ana</customer><item sku="A" qty="2">Pen</item><item sku="B">Ink</item><note/></order>')
print(json.dumps({etree.QName(x).localname: to_obj(x)}, indent=1))
y = etree.fromstring('<order id="8"><customer>Bo</customer><item sku="C">Cap</item></order>')
print(json.dumps(to_obj(y)))

{
 "order": {
  "id": "7",
  "customer": "Ana",
  "items": [
   {
    "sku": "A",
    "qty": "2",
    "value": "Pen"
   },
   {
    "sku": "B",
    "value": "Ink"
   }
  ],
  "note": null
 }
}
{"id": "8", "customer": "Bo", "items": [{"sku": "C", "value": "Cap"}]}

The second line is the test that matters. Order 8 has a single item, and items is still an array with one object in it. A consumer can loop over items without checking its type. The attributes are plain keys, the text is under value, and the code is the documentation of the mapping. Notice that qty is still the string "2": the function does not guess types. Add a type map keyed by attribute or element name, built from the schema, when you want numbers.

Better: generate from the schema

A hand-written list of array names drifts from the real format. If an XSD exists, derive the rules from it: every element with maxOccurs above one is a list, every xs:integer or xs:decimal becomes a number, every nillable element can be null. Code generators for XML bindings (JAXB, xsdata, xsd.exe) do this, and a binding serialized with a JSON library gives you the same shape every time. Trust a schema-driven mapping over a data-driven one.

Treat the JSON as a new contract

The converted JSON is a new interface with its own consumers. Give it a schema (a JSON Schema, see writing your first JSON Schema), version it, and test the conversion on documents that exercise every rule: zero, one and many items; empty and absent optional elements; attributes with and without text; an unusual character. Do not promise that it can be turned back into the original XML unless you tested a round trip, because details such as attribute order, comments and whitespace do not survive.

During a migration, you can serve both formats from one source of truth (keeping both XML and JSON output from one source), and retire the XML when the last consumer moves (migrating a legacy XML interface to JSON).

Testing the conversion

A conversion is code, and it deserves tests built from real documents rather than the tidy one in the documentation. Collect a corpus: the smallest valid document, the largest, one with every optional element present, one with none, and a few that came from production and surprised someone. Convert them all and keep the JSON next to the XML in the repository as expected output. A change to the mapping then shows up as a diff in files a reviewer can read, and an accidental change in shape is caught before a consumer sees it.

Add a property you can check automatically: every field that the mapping declares as a list is an array in every output, and every field that it declares as a scalar is never an array. A few lines of code walk the converted JSON and assert this against the declared sets, and they find the one-item bug in seconds, long before a customer finds it in the field.

Related reading

The cluster hub is designing XML documents that age well. Other conversion routes are in converting XML to JSON and converting JSON to XML with sensible element names. The xmltodict conventions are documented in its README, and JSON itself in RFC 8259. Convert a snippet in the XML to JSON tool.

When a generic converter is good enough

A one-off analysis. If you need the data in a notebook once, a generic converter and a look at the output is fine.

The data is regular. If every element has a fixed shape and you control the producer, the guesses never go wrong.

You need a lossless round trip. The @ and #text convention exists for reversibility. If you must rebuild XML from the JSON, accept its ugliness.

The consumer is another script of yours. Internal pipelines can adapt to whatever the converter emits. Public APIs cannot.