Dealing With XML Feeds That Break Their Own Schema

A feed has a specification, and the feeds you meet do not always follow it. The publisher's template emits a raw ampersand, a plugin leaves out an identifier, a date field contains the word "yesterday", a re-export duplicates an entry. The wrong responses are the two extremes: reject the whole feed because one entry is bad, or accept everything and let the bad data into your tables. The workable middle has three layers.

The defective feed

<feed xmlns="http://www.w3.org/2005/Atom"><title>Defects</title><id>urn:d</id><updated>2026-10-27T00:00:00Z</updated>
<entry><id>ok1</id><title>Fine</title><updated>2026-10-26T10:00:00Z</updated></entry>
<entry><title>No id</title><updated>2026-10-26T10:00:00Z</updated></entry>
<entry><id>bad-date</id><title>Bad date</title><updated>yesterday</updated></entry>
<entry><id>ok1</id><title>Duplicate id</title><updated>2026-10-26T11:00:00Z</updated></entry>
<entry><id>extra</id><title>Extra element</title><updated>2026-10-26T12:00:00Z</updated><weird>x</weird></entry>
<entry><id>amp</id><title>Smith & Sons</title><updated>2026-10-26T13:00:00Z</updated></entry>
</feed>

Six entries, five problems. A raw & in the last title makes the document not well-formed. The other defects (a missing id, a non-date updated, a duplicate id, an unknown element) leave it well-formed and break the Atom rules, which require id and a valid updated and forbid duplicates.

Layer 1: syntax

try: etree.fromstring(data)
except etree.XMLSyntaxError as e: print("strict ->", e.msg, "line", e.lineno)
p = etree.XMLParser(recover=True)
root = etree.fromstring(data, p)
print("recover -> parsed", len(root.findall("a:entry", A)), "entries; error_log:", [(x.line, x.message[:40]) for x in p.error_log][:2])

strict -> xmlParseEntityRef: no name, line 8, column 34 line 8
recover -> parsed 6 entries; error_log: [(8, 'xmlParseEntityRef: no name')]

The strict parse refused the document at line 8 with the usual message for a raw ampersand. The recover=True parse returned all six entries, and noted the error in p.error_log. That is the trap of recover mode: the result looks complete and the defect is recorded in a log that nobody reads. It is also unclear what the parser did with the broken part (it dropped the & and kept the rest, in this case; with a truncated tag it may guess differently). Use recover to look at a feed, or as a measured fallback when you check the log, never as a silent default.

The better approach for a known defect with one safe interpretation is to repair it explicitly and then parse strictly:

fixed = re.sub(rb"&(?!(amp|lt|gt|quot|apos|#\d+|#x[0-9a-fA-F]+);)", b"&", data)
root = etree.fromstring(fixed)

The regular expression escapes every & that is not already the start of an entity or character reference. A raw ampersand in text has only one reasonable meaning, a literal ampersand, so the repair is safe, and the document then parses strictly. Record that a repair happened (a counter and a log line), because it is a signal about the publisher, and because an unrepaired reference such as   still fails and needs its own decision (undefined entity errors in XML). Do not extend this to other syntax defects without thought. A mismatched tag, an unclosed element or a stray < has several possible intended structures, and guessing among them is how you load wrong data.

Layer 2: validate each entry

Once the document parses, validate entry by entry against the rules you rely on, and keep a list of rejects with reasons:

for n, e in enumerate(root.iterfind("a:entry", A), 1):
    i, t, u = (e.findtext(f"a:{k}", namespaces=A) for k in ("id", "title", "updated"))
    reason = None
    if not i: reason = "missing id"
    elif i in seen: reason = f"duplicate id {i}"
    else:
        try: datetime.fromisoformat(u.replace("Z", "+00:00"))
        except Exception: reason = f"bad updated {u!r}"
    ...

  entry 5 (extra) has unknown elements ['weird'] -> kept, counted
accepted: ['ok1', 'extra', 'amp']
rejected: [(2, None, 'missing id'), (3, 'bad-date', "bad updated 'yesterday'"), (4, 'ok1', 'duplicate id ok1')]

Three entries accepted: the good one, the one with the extra element and the one that needed the ampersand repair. Three rejected: entry 2 (no id), entry 3 (date yesterday) and entry 4 (the second ok1). Each reject is recorded with its position in the feed, its id if it has one and a specific reason.

The decisions embedded here are the ones to make explicitly. A duplicate id was resolved as "first wins". Other defensible rules: last wins (the later one is probably an update) or newest updated wins. The right rule depends on the publisher's behavior, and the point is that you chose. An unknown element was kept and counted. Extra elements in a feed are usually extensions, and rejecting an entry because it has more than you expected punishes the data for the sender's extra detail; counting tells you when a new element appears. A bad date was rejected here. The alternative is to accept the entry with a null date, keep it and flag it, which is better when the content is valuable and the date is only used for ordering. The right call depends on what the date does downstream.

Layer 3: report

A rejects table is the output of layer 2 that matters most:

With the raw bytes of the feed kept, any reject can be reproduced and sent to the publisher with the exact entry. Two aggregate numbers go with it: the reject rate for this run, and the same for the previous run. A feed that rejects 3 of 6 entries is a broken feed; one that rejects 3 of 50,000 may be an ordinary Tuesday. Set a threshold above which the whole run is held for review, because a sudden jump means the export changed and the repairs you wrote for the old defects will not match.

What you cannot repair

Know where to stop. Do not repair what has several readings: a truncated document (the missing part is gone), a date like 03/04/2026 whose order you cannot know, an encoding declared wrongly in a way that decodes without error but gives wrong characters (the declared-wrong-encoding case needs a decision about the real encoding, not a guess; see encoding mismatch errors in XML documents). For those, reject the entry or the feed and tell the publisher. A published feed that breaks its own schema at a rate you cannot absorb is a conversation to have with the publisher, with the rejects table as the agenda.

Prevention on your side

When you publish feeds, validate before you publish: a schema check, a check that ids are unique and present, dates parse. Build the XML with a library, so escaping is correct (sanitizing data you put into an XML document). The defects in this article are the ones that show up in feeds built by string concatenation.

Related reading

For validation against a published schema see validating XML against an XSD schema; for the parse errors involved, fixing XML parse errors fast; for the overview, working with XML feeds and APIs. The Atom rules for ids and dates are in RFC 4287, section 4.1.1 and 4.2.15, and the options of the lxml parser in its parsing guide. Test a path in the XPath tester.

When a feed is too broken to repair

Defects touch the structure. Missing closing tags, interleaved elements and truncation need the publisher.

The reject rate is high. Past a few percent on a feed that used to be clean, stop loading and ask.

The defect changes meaning. A feed that swapped price and cost is well-formed and valid and wrong. Compare distributions with the previous run.

The feed is an input you can replace. If there is an API or a better feed, move to it rather than maintain repairs for a bad one.