Detecting Structure Changes in an XML Feed Over Time

The loader ran. The job was green. The table has the right number of rows, and the price column is full of nulls since Tuesday. The feed's owner renamed price to cost, added a currency attribute and told nobody. Nothing failed, because nothing in your pipeline asserted that price has to exist: the XPath returned an empty node set, the code treated that as "no value", and the nulls went to the database. The cure is to look at the shape of every delivery and compare it with the last one, before loading.

Two deliveries

Monday's feed:

<feed><item id="1"><title>A</title><price>1.00</price></item><item id="2"><title>B</title><price>2.00</price><sale>true</sale></item></feed>

Tuesday's feed:

<feed><item id="1"><title>A</title><cost currency="EUR">1.00</cost></item><item id="2"><title>B</title><cost currency="EUR">2.00</cost><badge>new</badge></item></feed>

The records are the same two items. The price element became cost with a currency attribute, sale disappeared and badge appeared in one record. A loader keyed on item/price sees no price in any record. A person comparing the files by eye finds this in a minute. A person comparing two 400 MB files will not, and a feed delivered daily is a comparison nobody does.

A shape snapshot

The idea is to reduce each file to the set of paths it contains, with a count for each. A path is the chain of element names from the root, like /feed/item/price, and an attribute path adds /@name. Two feeds with the same paths and similar counts have the same structure, whatever the values. The script streams the file with iterparse so the size of the feed does not matter:

import sys
from collections import Counter
from lxml import etree

def shape(path):
    seen = Counter()
    for _, el in etree.iterparse(path, events=("end",)):
        parts = []
        node = el
        while node is not None:
            parts.append(etree.QName(node).localname)
            node = node.getparent()
        key = "/" + "/".join(reversed(parts))
        seen[key] += 1
        for a in el.attrib:
            seen[f"{key}/@{a}"] += 1
        el.clear()
    return seen

old, new = shape(sys.argv[1]), shape(sys.argv[2])
for k in sorted(set(old) | set(new)):
    if k not in old:
        print("added  ", k, new[k])
    elif k not in new:
        print("removed", k, old[k])

Run it on the two files:

python3 shape.py mon1.xml mon2.xml

added   /feed/item/badge 1
added   /feed/item/cost 2
added   /feed/item/cost/@currency 2
removed /feed/item/price 2
removed /feed/item/sale 1

Five lines, and they are the complete story. price and sale are gone, cost, its attribute and badge are new. The numbers after each path are the counts in the file that has it: two cost elements, one badge. The script uses the local name of each element, so namespace prefixes do not create false differences. If the same local name appears in two namespaces and the difference matters, build the key from the full qualified name instead.

One detail in the code is worth explaining. el.clear() after each element keeps memory flat, because without it iterparse keeps the whole tree it has built. The ancestors are still reachable through getparent() while the element ends, which is why the path can be built at the end event. For a very large file also delete the processed siblings; the memory recipes in parsing large XML files with iterparse show how.

Counts, not only names

The script above reports only paths that exist in one file and not the other. Many real breakages leave the path in place and change how often it occurs. If shipping appeared in 98 percent of records last week and in 4 percent today, the element exists in both files and the feed has still changed, usually because a source system stopped filling it. Extend the comparison with a ratio check:

for k in sorted(set(old) & set(new)):
    if old[k] and abs(new[k] - old[k]) / old[k] > 0.2:
        print("count  ", k, old[k], "->", new[k])

Compare counts relative to the number of records rather than absolute counts, since a file can legitimately contain more or fewer records. Divide each count by the count of the repeating element, /feed/item here, and flag a path whose proportion moved by more than a threshold you choose. Twenty percentage points is a reasonable start for a feed whose fields are mostly mandatory, and you will adjust it after a few weeks of false alarms.

Where to run it

Run the comparison as the first step of the load, before parsing records into tables. Store the snapshot of each accepted delivery as a small JSON file next to the data, or a table of path and count with a delivery date. Compare each new file to the last accepted snapshot, not to the one before it, so a slow drift cannot slip through in small steps. Stop the load and notify when anything is added or removed, or let additions through with a warning when your code reads only named fields, since a new element does not break a loader that never asks for it. Removals deserve a hard stop, because a removal is exactly what produces silent nulls.

Pair it with a schema. A snapshot tells you the shape changed, and a validation tells you if the new shape is acceptable (validating XML against an XSD schema). When no schema exists, testing a feed's compatibility before writing a schema describes how to infer one from samples, and the same shape counts can seed it.

Making the alert useful

A comparison that cries wolf every day gets muted. Three habits keep it useful. Print the delivery date and the file name with the result, so whoever reads the alert knows which file to open. Group the output by kind, removals first, since those are the ones that break loaders. And keep an allowlist of paths you have decided to ignore, such as a tracking element that the provider changes on every release, so the same harmless difference is not reported twice.

When the report is clean, say so. A line that reads "shape unchanged, 14 paths, 2,310 items" in the job log is evidence that the check ran. Without it, a silent success and a check that never executed look the same, and you only find out which one it was after the next incident.

Related reading

The cluster hub is comparing and versioning XML. For feeds that break their own contract, see dealing with XML feeds that break their own schema, and for a scheduled check, checking feed health with a daily script. The iterparse API is documented on the lxml parsing page, and the idea of inferring a structure from samples is the same one behind XML Schema inference tools. Test the XPath your loader depends on with the XPath tester.

When a path snapshot cannot catch the change

The meaning changed and the shape did not. A price element that switches from euros to cents has the same path and a different value. Add range checks on values, such as a minimum and maximum per numeric field.

The change is in the text of a code list. A status element that starts sending PAID instead of paid is invisible to a path comparison. Track the distinct values of low-cardinality fields.

Optional elements that appear rarely. A path seen once in a million records may be absent from a small delivery without meaning anything. Judge rare paths over several deliveries.

The feed is genuinely heterogeneous. If records of different kinds share one feed, the shape is a union and changes in one kind hide behind the others. Take snapshots per record type.