Testing Feed Compatibility Before a Schema Change Ships
Which XSD edits break old documents, which break old consumers, and a script that tests six changes both ways against saved samples before you ship.
Changing the schema of a feed is a deployment with two sides. The producer starts sending new documents, and the consumers keep reading with whatever validator and parser they have. A change that looks harmless in the schema file can reject every document a consumer receives the next morning, or reject every document the producer used to be able to send. You can find out which before you ship, with a script and a folder of saved samples.
Two directions
Compatibility has two meanings, and people usually test one.
Backward compatible: documents written to the old schema still validate against the new one. This protects the consumers who upgraded and the old archives you will have to read again.
Forward compatible: documents written to the new schema still validate against the old schema. This protects the consumers who have not upgraded yet, and in practice they are most of them for a while.
A change can be one without the other, and some are neither. If both sides of a feed deploy at the same instant you could ignore the second, but they never do.
The six changes
I made six edited copies of the schema, each representing a change someone will propose:
Add an optional element (
coupon,minOccurs="0").Add a value to an enumeration (
voidto the status list).Make an optional element required (
note).Tighten a pattern (SKU digits from three to four).
Add a required element (
coupon).Loosen a pattern (SKU digits from exactly three to three to five).
For each one, the test needs a document the new schema would produce, written by hand once, for example the optional-element change gets an order with a coupon. Then:
from lxml import etree
old = etree.XMLSchema(etree.parse("shop.xsd"))
corpus_old = [etree.parse(f) for f in ("good.xml",)] # saved real documents
new_docs = {"v2_optional_element": b"<orders><order id='1'>...<coupon>X</coupon>...</order></orders>", ...}
print(f"{'change':22} {'old docs pass new XSD':22} {'new docs pass old XSD'}")
for name, doc in new_docs.items():
new = etree.XMLSchema(etree.parse(name + ".xsd"))
back = all(new.validate(d) for d in corpus_old)
fwd = old.validate(etree.fromstring(doc))
print(f"{name:22} {str(back):22} {fwd}")
if not back:
print(" ", new.error_log[0].message)What it printed
change old docs pass new XSD new docs pass old XSD
v2_optional_element True False
v2_new_enum_value True False
v2_required_note False True
Element 'item': This element is not expected. Expected is ( note ).
v2_stricter_sku False False
Element 'item', attribute 'sku': [facet 'pattern'] The value 'A-100' is not accepted by the pattern '[A-Z]-[0-9]{4}'.
v2_required_coupon False False
Element 'item': This element is not expected. Expected is one of ( note, coupon ).
v2_looser_sku True False(The test files use full documents; the dots above stand in for the unchanged children.) Read the rows:
| Change | Old documents still valid | Old consumers accept new documents |
|---|---|---|
| Optional element added | yes | no |
| Enumeration value added | yes | no |
| Optional made required | no | yes (new docs include it) |
| Pattern tightened | no | no, for new-style values |
| Required element added | no | no |
| Pattern loosened | yes | no, for new-style values |
The result worth taking away is the first, second and last rows: three changes that every guide calls "safe" are safe only in the backward direction. They protect old data and break old consumers, the moment the producer starts using the new freedom. Adding a coupon to the schema breaks nothing until someone sends one, at which point every consumer still on the old schema rejects the order, with This element is not expected. The same goes for the new void status and for SKUs of five digits.
The third row is the mirror: making note required cannot hurt a consumer who sees only new documents, because the new documents all have it, and it breaks every old document and every sender that has not changed. The fourth and fifth rows are both ways bad. They are the changes that need a new version of the format.
What to do with the table
Prefer changes that are safe both ways, which are few. Relaxing a constraint in a way old consumers ignore, such as an xs:any extension point in the old schema from the start, is what makes the first and second rows harmless. If your schema ships with a wildcard <xs:any namespace="##other" processContents="lax" minOccurs="0" maxOccurs="unbounded"/> at the end of each element, additions in another namespace pass old validators.
Order the deployment. For a backward-compatible change (rows 1, 2, 6), upgrade the consumers first, then the producer. For rows 3, 4 and 5, the order does not help: a version bump is required, with both versions accepted in parallel for a transition period.
Version in a visible place. An attribute on the root (version="2") or the namespace (urn:shop:v2) lets a consumer choose the right schema. A new namespace per breaking version is the cleanest, because the old schema rejects the new documents loudly, as a different element, instead of confusingly, as a mismatch inside the order.
Test with real documents. The corpus is the point. A year of saved feeds will contain empty notes, odd SKUs and a status somebody typed by hand, and the new schema meets all of them in the test, not in production. Save every feed file you receive for at least a month, with its date, in cheap storage.
Putting it in a pipeline
The script is small enough to run on every change to the schema directory in CI. The inputs are the old schema (from the main branch), the new one (from the branch), the corpus of saved documents, and a handful of hand-written documents for the new behavior. The outputs are the table and a non-zero exit code if any "old documents pass" is false, unless the change is marked as a breaking version. Keep it dumb: the aim is that nobody merges a schema change without having seen the table.
Add a check on the other consumers you can reach. If you own a parser that reads the feed with a schema-derived binding (JAXB, xsd.exe classes), regenerate the code from the new schema and run its tests; generated classes break in their own ways, such as a field that becomes required.
Related reading
For the mechanics of validation see validating XML in Python with lxml and XSD and common XSD validation errors explained. For the schema itself, writing a basic XSD schema, and for the whole topic, validating XML against an XSD schema. Feeds that break their own schema are the subject of dealing with XML feeds that break their own schema. Extension points and versioning are discussed in the W3C's versioning of XML languages note, and the wildcard syntax in the XML Schema structures specification. Try a path on a sample with the XPath tester.
When a schema change needs a new version
The change breaks both directions. Tightened patterns, new required elements, renamed elements and changed types all fail someone. Publish a new version beside the old, announce a date for retiring the old one, and accept both until then.
The meaning of a field changes. A field that was net price and becomes gross price validates against both schemas and silently corrupts every total. No schema test finds this. Rename the field.
Consumers cannot be reached. If you do not know who reads the feed, assume someone does and every change is breaking. Add a version in the URL and keep the old one running.
The corpus is too small. A compatibility test on three documents tests three documents. If the feed is critical and the corpus is thin, collect more before changing anything.
