Designing XML Documents That Age Well From Day One

Every XML format starts as a convenience. Someone needs to get an order, a catalog or a configuration from one program to another, and they write a few elements. It works. A year later a second consumer arrives, then a partner, then a requirement nobody thought of. The format that was a convenience is now a contract with readers you cannot call, and every change is a negotiation. This guide is about the decisions that make that negotiation easier, taken in the order you would take them when designing a format. It collects the cluster's detailed posts into one path and adds a worked example that was run end to end.

What it means for a format to age well

A format ages well when the change you will need in two years is cheap. You do not know what the change is. You know its shape, though, because the same few pressures arrive for every format: a new field somebody wants, a field that turns out to need structure, a consumer that reads it in a language you did not plan for, a partner who wants to add their own data, a mistake in the original design that must be corrected, and growth in size. A format ages badly when each of these forces a break.

Three parties share the cost of a change, and thinking about them separately is useful.

Writers produce documents. They are usually few and often under your control. A writer change is cheap because you can deploy it.

Readers consume documents. They are many, and some are not under your control. A reader that works today and fails after your change is the expensive case, and the whole discipline of format design is about not causing it.

The schema is the written agreement between them. It is also software, with versions, and it is the one place where the rules for change can be stated and checked mechanically.

Everything else in this guide follows from a single principle: make additions free and make breaks loud. An addition should cost readers nothing, because they ignore it. A break should be impossible to miss, because the document fails validation in the reader's first step, instead of producing wrong data three steps later. The sections below show how each design choice moves a format toward or away from that principle.

Five questions before the first element

Answer these in writing before choosing a single name. They take twenty minutes and save months.

1. Who reads this, in which languages? If the readers are Java services and a Python notebook, field names should map cleanly to identifiers in both. If one reader is a spreadsheet user, deeply nested structures are a burden.

2. How long will documents live? A message that is consumed in a second can follow the current version of the format. An archive that must be readable in ten years needs a version marker, a published schema and documentation that does not depend on someone's memory.

3. Who writes it, and can they be told to change? One producer you control is a different problem from forty partners who each implemented the spec their own way.

4. How big will documents get? A kilobyte document and a gigabyte document call for different structures. Streaming readers need a repeating element at a predictable depth, and the largest file you will ever see should shape the design from the start.

5. Is there an existing format that already does this? If a standard fits, such as Atom, UBL, SVG or Office Open XML, use it, even imperfectly. Every standard comes with tooling and readers you would otherwise have to build.

If the answers make XML look wrong, say so now. Choosing XML over the alternatives, or not goes through that decision in detail. For the rest of this guide, assume XML was the right choice.

Names and casing

Names are the first thing a reader sees and the last thing you can change. XML names are case sensitive, must start with a letter or underscore, and can contain letters, digits, hyphens, underscores and periods. Spaces and colons are out, except for namespace prefixes. Names beginning with xml in any case are reserved.

The rules leave a lot of room, and the decision that matters is how names turn into code. A name with a hyphen, like order-total, is not a valid identifier in most languages, so every code generator and every hand-written mapping must rename it. A name in camelCase or snake_case becomes a field with no change. Choosing XML element names and casing compares the conventions and gives a decision procedure. The short version: pick one convention for elements and attributes both, match the dominant consumer's language, and never encode data in a name (item1, item2) because a schema cannot describe it.

Name for the content, not the container. total says what it is. decimalValue1 says how it is stored. Abbreviate only what your domain abbreviates, such as sku or iban. Keep singular names for items and plural names for wrappers: parcels holds parcel elements.

Attribute or element

This is the choice people argue about, and it has a straightforward answer for most cases. An attribute fits a short, single, atomic value that describes the element: an id, a type, a unit, a language. A child element fits everything that is long, repeatable, structured or likely to grow.

Two facts break ties. First, attribute values are whitespace-normalized by the parser: a line break or tab in an attribute comes back as a space, and in an element it survives. That is why free text belongs in elements. Second, an attribute cannot grow. A weight stored as weight="12" has no place for a unit, and adding weightUnit="kg" as a sibling attribute detaches the unit from the value. Starting with <weight unit="kg">12</weight> costs a few bytes and avoids the rename. When attributes are the better choice shows the whitespace experiment and the conversion effects.

Adding an attribute is backward compatible for readers that ignore unknown attributes. Turning an attribute into an element later is a break for every reader that looked at the attribute. If a value might need structure, make it an element on day one.

Lists and optional data

XML has no list type, so a list is a repeated element, and an optional value is an element that may be missing. Both cause trouble at the edges, in code that converts the document to something else.

For lists, wrap the repeated items in a plural container: <parcels><parcel/><parcel/></parcels>. The wrapper lets the document say "there are no parcels" (an empty wrapper) separately from "parcels were not provided" (no wrapper), gives the list a place for its own attributes such as a count, and keeps the parent's other children from being mistaken for items. A bare repeat is fine when the parent holds nothing else.

For optional data, decide what absent, empty and nil mean and write it down. My default: absent means not provided, empty means an empty string, and xsi:nil="true" is for elements that must be present but have no value. Avoid sentinel values like N/A or -1, which end up in numeric columns.

The real danger is in conversion. A generic XML-to-JSON converter that guesses types from the data turns a one-item list into a scalar and a many-item list into an array, so one field has two types depending on how many items an order happened to have. Fix it by declaring which elements are lists, from the schema. Modeling lists and optional data in XML has the experiment, with the exact output.

Binary data and large payloads

XML cannot carry raw bytes. Control characters other than tab, line feed and carriage return are forbidden even as character references, and arbitrary bytes are not valid in any text encoding. Binary content must be encoded as text, almost always Base64, declared in a schema as xs:base64Binary.

The cost is measurable. Base64 grows data by one third: 1 MB of random bytes became 1,333,336 characters in my test, and hex doubled it exactly. A DOM parser holds the encoded text and the decoded bytes, so a 100 MB attachment needs more than 200 MB of memory. Past a few hundred kilobytes, send a reference (a URL, an id and a hash), a package such as a zip with a manifest, or MTOM if you are in SOAP. Embedding binary data in XML has the numbers and the alternatives.

Design the document so that the large thing is not inside the part everyone reads. A header with the metadata, followed by a payload element that readers can skip, lets a consumer inspect a document without decoding its attachment.

Mixed content

Mixed content is an element holding both text and child elements: a paragraph with bold words. It is essential for documents and damaging for data. A parser exposes the pieces as the element's text and each child's tail, so a sentence is spread over several places, and mapping it to a record or to JSON loses either the markup or the order.

For a data format, forbid it: containers hold elements, leaves hold text, never both. If a record needs a rich text field, give it its own leaf element with a declared type, such as an HTML fragment, and parse it separately. If you are designing a document format, embrace mixed content and use document tools, XSLT above all. Mixed content in XML and why it hurts parsing shows the text and tail mechanics.

Namespaces and extension points

A format that others will extend needs a place for extensions that cannot collide with the core. Namespaces provide it. Your core vocabulary lives in one namespace, and every extender uses their own. A reader selects the core by namespace URI and ignores the rest. Matching is by URI, not by prefix, so a document that renames o: to ord: still works.

State the policy for unknown elements: ignore, preserve and pass on, or reject. Ignore is the default for exchange. Preserve is for intermediaries that forward documents. Reject is for security-sensitive cases such as signed documents. Express the extension point in the schema with xs:any namespace="##other". Using XML namespaces for extensions without the chaos walks through a reader and the three policies.

A schema without an extension point is closed, and a closed schema turns every addition into a break for readers that validate. That is the single most common reason a "backward compatible" change broke somebody.

Versioning

Plan for versions before the first release. Two markers do different jobs. A version attribute on the root is advisory and cheap: readers can log it and branch if they need to, and it should change with every release that adds something. The namespace URI is a hard break: a reader looking for elements in the version 1 namespace finds none in a version 2 document and fails at once. Use it only for changes old readers must not attempt to read.

Write the rules down:

1. Additions are optional elements or attributes, appended to sequences or placed in the extension namespace.

2. Readers must ignore unknown elements and attributes.

3. Removing or renaming something needs a deprecation period during which both forms are written.

4. A change in meaning gets a new name. Never reuse an old one.

5. The version attribute changes with every release; the namespace changes only on a break.

6. Every version's schema is published and kept.

In my test, a strict schema rejected a version 2 document that added one currency element, with "This element is not expected". The same document, with the new element in another namespace and an xs:any extension point, validated. The first schema turned an innocent addition into an outage, and the second made it free. Versioning an XML format without breaking readers shows both outputs.

A worked design: a shipment format

Here is the whole thing applied to a small example. The requirement: a warehouse sends shipments to carriers, each with a reference and one or more parcels, each parcel with a weight and an optional note. Carriers will want to add tracking data of their own.

The decisions, using the rules above. Elements and attributes are lower camel case, which here means single lowercase words. The shipment has a version attribute (identity and state go in attributes). parcels wraps the repeated parcel elements. Each parcel has an id attribute, a weight element with a required unit attribute so the unit cannot be detached, and an optional note. The vocabulary lives in the namespace urn:example:shipments:1. After the parcels, the schema allows any elements from other namespaces.

<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
           xmlns="urn:example:shipments:1" targetNamespace="urn:example:shipments:1"
           elementFormDefault="qualified">
  <xs:element name="shipment">
    <xs:complexType>
      <xs:sequence>
        <xs:element name="reference" type="xs:string"/>
        <xs:element name="parcels">
          <xs:complexType>
            <xs:sequence>
              <xs:element name="parcel" maxOccurs="unbounded">
                <xs:complexType>
                  <xs:sequence>
                    <xs:element name="weight">
                      <xs:complexType>
                        <xs:simpleContent>
                          <xs:extension base="xs:decimal">
                            <xs:attribute name="unit" type="xs:string" use="required"/>
                          </xs:extension>
                        </xs:simpleContent>
                      </xs:complexType>
                    </xs:element>
                    <xs:element name="note" type="xs:string" minOccurs="0"/>
                  </xs:sequence>
                  <xs:attribute name="id" type="xs:string" use="required"/>
                </xs:complexType>
              </xs:element>
            </xs:sequence>
          </xs:complexType>
        </xs:element>
        <xs:any namespace="##other" processContents="lax" minOccurs="0" maxOccurs="unbounded"/>
      </xs:sequence>
      <xs:attribute name="version" type="xs:string" use="required"/>
    </xs:complexType>
  </xs:element>
</xs:schema>

A valid version 1 document with two parcels, one with a note:

<shipment xmlns="urn:example:shipments:1" version="1">
  <reference>SH-1001</reference>
  <parcels>
    <parcel id="P1"><weight unit="kg">2.5</weight></parcel>
    <parcel id="P2"><weight unit="kg">0.8</weight><note>fragile</note></parcel>
  </parcels>
</shipment>

A carrier adds a tracking element in its own namespace, after the parcels, and bumps the version attribute to 1.1:

<shipment xmlns="urn:example:shipments:1" xmlns:t="urn:example:tracking" version="1.1">
  <reference>SH-1002</reference>
  <parcels>
    <parcel id="P1"><weight unit="kg">2.5</weight></parcel>
  </parcels>
  <t:tracking carrier="DHL">JD0146000123</t:tracking>
</shipment>

Validate both, and three documents that break the rules in different ways:

for f in s1 s2 s3 s4 s5; do xmllint --noout --schema ship.xsd $f.xml; done

s1.xml validates
s2.xml validates
s3.xml:3: element parcels: Schemas validity error : Element '{urn:example:shipments:1}parcels': Missing child element(s). Expected is ( {urn:example:shipments:1}parcel ).
s3.xml fails to validate
s4.xml:4: element weight: Schemas validity error : Element '{urn:example:shipments:1}weight': The attribute 'unit' is required but missing.
s4.xml fails to validate
s5.xml:2: element parcels: Schemas validity error : Element '{urn:example:shipments:1}parcels': This element is not expected. Expected is ( {urn:example:shipments:1}reference ).
s5.xml fails to validate

The first two pass, including the extension, which is the point of the design: the carrier's addition cost the core readers nothing. The other three are the loud failures we want. s3.xml has an empty parcels wrapper and fails, because the schema says at least one parcel. Whether that is right is a business decision, and this is the moment to make it: a shipment with no parcels may be meaningless, or it may be a draft. If drafts are allowed, the schema needs minOccurs="0" on parcel, and the example shows how a test document finds the question. s4.xml is missing the unit, which is exactly the mistake the required attribute prevents. s5.xml puts parcels before reference, which a sequence forbids. If your writers will reorder elements, use xs:all or accept order in the reader, and say so.

Two more design observations from the exercise. The error messages name the element and what was expected, so they serve as the first line of an error report to a partner. And the validation took a single command, so it is cheap to run in a build or an intake service.

Designing for large documents

A format that works at a kilobyte can fail at a gigabyte, and the structure decides which. A streaming parser reads a document once, front to back, in constant memory, and it can only do that when the repeating records sit at a predictable depth and each record is self-contained. Put the records in one repeating element directly under the root or one wrapper, give every record its own identifier, and avoid references that point forward to a definition later in the file. A header that appears before the records, holding the metadata a reader needs to interpret them, lets a consumer start work after the first few kilobytes.

Avoid designs where one record needs another to be understood, such as a list of ids at the top and the detail at the bottom. A reader then holds the first part while it waits for the second, and the memory advantage of streaming disappears. If two parts must be related, give both an id and let the reader join them later, in a database, rather than at parse time.

Think about splitting as well. A format that can be cut at record boundaries into several valid files, each with its own root and header, lets a consumer process parts in parallel and retry one part without redoing the rest. Document the rule: what the header carries, how many records a part may hold and how a reader reassembles the whole. Splitting and merging are covered in splitting a big XML file by repeating element and merging several XML files into one.

Compression changes the arithmetic. XML repeats element names on every record, which is why it compresses well: a gzip of a large feed is usually a small fraction of the original. Do not shorten names to save space. Compress in transport and keep the names readable (compressing XML archives and reading them in place covers the reading side).

Error handling is part of the design

A format is not finished until it says what happens to a bad document. Decide whether a reader rejects the whole file or the one bad record, and write it in the specification. All-or-nothing is simple and right for financial data, where a partial import is worse than none. Record-by-record is right for a large feed where one bad product should not block ten thousand good ones, and it needs a place to report the rejects. Give writers the means to learn about errors: a response document that lists each failure with the record id and the validation message turns a mystery into a ticket they can fix themselves.

A review checklist

Run this against any format, new or inherited. Each item is a question with a failure mode behind it.

• Is every name a valid identifier in the languages that consume it? If not, expect a renaming layer.

• Are free-text values in elements, not attributes? Attributes normalize whitespace.

• Does every number have a unit or a currency where it could be ambiguous? Put it in an attribute on the element, not in the name.

• **Is every repeated thing wrapped, and does the schema state minOccurs and maxOccurs?** Defaults are easy to misread.

• Are absent, empty and nil defined? One sentence in the specification is enough.

• Is there an extension point, and what is the policy for unknown content? A closed schema is a future outage.

• Is there a version marker, and a written rule for when it changes?

• Are large or binary values separable from the metadata? A reader should be able to skip them.

• Does any element mix text and children? If it is a data format, remove it.

• Is there a sample for every shape the schema allows? No sample, no confidence.

A test plan: the corpus

A format is software, and it needs tests. The test suite is a corpus of documents plus an expected result for each.

Collect, per version of the format: the smallest valid document; a document with every optional element present; a document with none; documents with zero, one and many of every repeated element; a document with the longest values allowed; documents with characters that need escaping and non-ASCII text; and a few invalid documents, each wrong in one way, with the error you expect. Add every real document that ever caused an incident.

Run the corpus on every change. Each valid old document must validate against the new schema, and each invalid one must still fail. A change that makes an old valid document invalid is a break, whatever the commit message says. That single check is the most valuable automation a format can have, and it is a ten-line script around xmllint.

Keep a second set for conversions. If the document is converted to JSON, to a table or to an object, store the expected output beside the input, so the conversion's behavior on a one-item list or an empty optional is pinned down (moving from XML to JSON gracefully shows a mapping and the tests it needs).

When two documents should be equal, compare them in canonical form rather than as bytes (canonical XML and why diffs need it). It stops a test from failing because of attribute order or indentation.

Habits for a format that keeps growing

Write the specification as you go. A schema says what is valid and not what it means. A page of prose for each element, with an example and the reason it exists, is the part that survives staff changes.

Add, do not repurpose. Reusing status with new values is a break for every reader that switch-cases on it. A new element for the new meaning is compatible.

Deprecate with a date. Mark the old element deprecated in the schema's documentation, write both forms during the period, log who still reads the old one if you can, and remove it on the announced date.

Keep old schemas. Consumers pinned to version 1 need version 1's rules, not a description of them.

Measure the readers. If you can, count which versions and which optional elements show up in traffic. A field nobody sends is a candidate for removal. A version still in use is a reason to wait.

Review diffs of the schema, not just of the code. A schema change is a contract change. Treat the diff of the XSD with the seriousness of a diff to a public API, because it is one. Comparing and versioning XML covers the tools for seeing what changed.

Moving off XML when the time comes

Sometimes the right move is to leave. A format whose readers have all become JavaScript, or a service that wants a compact binary encoding, may be better served elsewhere. Do it deliberately. Convert with an explicit mapping, not a generic converter. Serve both formats from one source of truth for as long as consumers need (keeping both XML and JSON output from one source), measure who still reads the old one, and retire it on a date (migrating a legacy XML interface to JSON).

A format designed with the rules in this guide is easier to migrate, too. Clean names, wrapped lists, typed values and a written specification turn the mapping into a table rather than an archaeology project.

Related reading

The earlier posts in this cluster cover the comparison side: canonical XML, diffing while ignoring attribute order and detecting structure changes in a feed. Validation is in validating XML against an XSD schema, with the commands in validating XML against XSD with xmllint. For when the conversation turns to alternatives, read XML versus JSON versus YAML for configuration.

The sources behind these rules are worth reading directly. The base specifications are XML 1.0 and Namespaces in XML, schemas are defined in XML Schema Part 1 and Part 2, and the W3C TAG's versioning finding is the best short treatment of compatibility in formats. Format sample documents while you design with the XML formatter, and test selectors with the XPath tester.

When this advice does not apply

You do not control the format. If the schema belongs to a standard body or a partner, you implement it as it is. Your job is a tolerant reader and good error messages, not a better design.

The document is a throwaway. A file that lives for an afternoon between two scripts of yours can use whatever names are quickest. Do not carry ceremony into scratch work.

The format is a document format. For prose, such as manuals and books, the advice about mixed content is reversed. Use the established vocabularies (DocBook, DITA, TEI, XHTML) and their tooling, and put your effort into the style sheets.

Size and speed dominate. A format that carries a million records a second may belong in a binary encoding. XML's readability is the thing you pay for, and if nobody reads the bytes, the payment buys nothing. See XML versus Protocol Buffers for services.

The cost of a break is genuinely zero. If there is one reader and you ship it together with the writer, the compatibility machinery is overhead. Add it when the second reader appears, and expect to pay for the late arrival.