Modeling Lists and Optional Data in XML Without Surprises

Lists and optional values are where XML models go wrong most often, and the mistakes show up in a different place from where they were made. The document is fine. The code that converts it to a table, an object or JSON is the one that breaks, usually on the first record with an unusual shape. A model that makes these cases explicit costs a few extra tags and saves the debugging.

Lists: repeat the element, wrap the repeats

XML has no list type. A list is an element that occurs more than once. There are two layouts. The bare form repeats the item directly under the parent:

<order id="7">
  <item sku="A"/>
  <item sku="B"/>
</order>

The wrapped form adds a container for the repeats:

<order id="7">
  <items>
    <item sku="A"/>
    <item sku="B"/>
  </items>
</order>

The wrapper costs two tags and buys three things. It expresses "the list is empty" (<items/>) separately from "the list is not given" (no items at all). It gives the list a place to carry its own attributes, such as count or truncated="true". And it keeps the parent's other children from being confused with items, which matters when the schema allows several kinds of repeated child. The bare form is shorter and fine when the parent holds only items, as in a feed's channel or a table's rows. For a document that other people will extend, I prefer the wrapper.

Use a plural wrapper and a singular item (items/item, addresses/address) and do not mix other element kinds into the wrapper. A wrapper that sometimes holds item and sometimes service is two lists pretending to be one.

The one-item trap

The structure is not where the surprises are. They are in the conversion. A converter looking at <order><item>a</item></order> has no way to know that item is a list in the schema, and it guesses from the data: one element is a value, two or more are a list. The result, with xmltodict:

import json, xmltodict
one = "<order><item>a</item></order>"
two = "<order><item>a</item><item>b</item></order>"
print(json.dumps(xmltodict.parse(one)), json.dumps(xmltodict.parse(two)))
print(json.dumps(xmltodict.parse(one, force_list=("item",))))

{"order": {"item": "a"}} {"order": {"item": ["a", "b"]}}
{"order": {"item": ["a"]}}

The same element is a string in one document and an array in the other. Code that loops over order["item"] iterates over the characters of the string for a one-item order, or fails with a type error, and it only happens for orders with a single item. That is the typical production bug: the test data had three items, and real life has a lot of orders with one. The fix is force_list, which tells the converter which names are lists whatever the data shows. Other tools have equivalents (isArray in fast-xml-parser, @XmlElementWrapper in JAXB). The better fix is to generate the list handling from the schema, so the declaration lives in one place (moving from XML to JSON gracefully covers the mapping conventions).

Optional: absent, empty and nil

Three states look alike and mean different things:

• Absent: the element is not there. The value is unknown, not applicable or not provided.

• Empty: the element is there, with no text: <note/> or <note></note>. For a string field this can mean "the empty string".

• Nil: the element is there and explicitly null. XML Schema Instance defines xsi:nil="true" for this, and a schema must declare the element nillable.

How a parser reports them:

from lxml import etree
doc = '<p xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><a/><b xsi:nil="true"/></p>'
r = etree.fromstring(doc)
print("a text:", repr(r.find("a").text), "b text:", repr(r.find("b").text), "c:", r.find("c"))

a text: None b text: None c: None

Here an empty element and a nil element both give None for their text, and an absent element gives None from find. To tell them apart you have to look at the node, not the text: test for find(...) is None to catch absence, and read the xsi:nil attribute to catch nil. Many loaders collapse all three into SQL NULL, which is often right and sometimes lossy. If your model needs the distinction, say so in the schema and keep it in the load step.

Pick one convention per field and write it down. The one I use: **absent means not provided, empty means an empty string, and xsi:nil is used only when a schema requires the element to be present but allows no value.** Avoid sentinel strings such as N/A, NULL or -1 in values. They end up in a numeric column and break it, or in a report and look like data.

Declaring it in a schema

An XSD expresses these with minOccurs, maxOccurs and nillable:

<xs:element name="item" type="ItemType" minOccurs="0" maxOccurs="unbounded"/>
<xs:element name="note" type="xs:string" minOccurs="0"/>
<xs:element name="discount" type="xs:decimal" nillable="true"/>

The first is a list that may be empty, the second an optional string, the third a decimal that must appear but may be nil. A schema that says these things lets a code generator emit a list type, a nullable type and a required-but-nullable type instead of three strings (writing a basic XSD schema has the declarations in context).

A test document for every shape

Before a model ships, write one test document for each shape it allows and load all of them through the real pipeline. For a list, that is zero items, one item, two items and a very large number. For an optional field, that is present with a value, present and empty, nil, and absent. For a nested optional structure, such as an address with an optional second line, that is each combination of the parts. Ten small files catch the one-item bug, the empty-string-versus-null bug and the missing-wrapper bug on the first run.

Keep the files with the schema. When the format changes, they are the regression suite, and when a new consumer arrives, they are the quickest description of what the format actually allows.

Related reading

The cluster hub is designing XML documents that age well. On the loading side, see handling optional XML elements and flattening repeating XML elements into rows. Nil is specified in XML Schema Part 1, and the force_list option in the xmltodict documentation. Try a conversion in the XML to JSON tool.

When the wrapper element is not worth it

The parent holds one kind of child. A feed's channel of item elements, or a table of row elements, does not gain from a wrapper. The parent is already the list.

You must match an existing format. RSS has no items wrapper, and adding one makes the feed invalid. Follow the format.

The list can never be absent or annotated. If an empty list and no list mean the same thing and the list never needs a count or a flag, a bare repeat is fine.

The payload is huge. For multi-gigabyte files, an extra level of nesting changes nothing measurable. Do not choose layouts to save tags.