Checking Feed Health With a Daily Script and Clear Alerts

A feed you depend on will break, and the way you find out matters. The usual way is a user: "the product list has not changed in a week". A daily health check finds it first, if it tests the right things. A status code is not enough, because the failures that matter return 200.

What to check, in order

The checks go from cheapest to most specific, and each failure is a different problem with a different owner.

1. Fetch completes within a timeout. A hung endpoint is its own failure.

2. HTTP status is 200 (or 304 for a conditional check).

3. Content type looks like XML. A maintenance page that returns HTML with status 200 is the commonest false pass.

4. Well-formedness. The body parses.

5. Item count meets a minimum. An empty feed is valid and useless.

6. Freshness. The newest item is younger than a threshold for this feed.

Add validation against a schema and a check of the item fields if you have a contract (validating XML against an XSD schema). The six above need nothing but the feed.

The script

def check(url, max_age_days=14, timeout=2, min_items=1):
    r = {"url": url, "status": None, "problems": []}
    t = time.time()
    try:
        resp = urllib.request.urlopen(urllib.request.Request(url, headers={"User-Agent": "dh-feed-health/1.0"}), timeout=timeout)
        body = resp.read(); r["status"] = resp.status; ctype = resp.headers.get("Content-Type", "")
    except urllib.error.HTTPError as e:
        r["status"] = e.code; r["problems"].append(f"http {e.code}"); return r
    except Exception as e:
        r["problems"].append(f"fetch failed: {type(e).__name__}"); return r
    r["bytes"] = len(body)
    if "xml" not in ctype: r["problems"].append(f"content-type {ctype!r}")
    try: root = etree.fromstring(body)
    except etree.XMLSyntaxError as e:
        r["problems"].append(f"not well-formed: {e.msg[:40]}"); return r
    items = root.findall("a:entry", A); r["items"] = len(items)
    if len(items) < min_items: r["problems"].append("no items")
    ...
    if age > max_age_days: r["problems"].append(f"stale: newest item {age} days old")
    return r

The function returns a dictionary with the observed values and a list of problems. It never raises: every failure becomes a problem entry, so one broken feed cannot stop the check of the others. It records what it saw (status, bytes, items, age) even on success, so you can chart them. Note the User-Agent: identify the monitor so the publisher knows who is polling.

The scenarios

I ran it against eight routes on the test server:

OK    {"status": 200, "problems": [], "seconds": 0.03, "bytes": 688, "items": 4, "newest_age_days": 0} atom.xml
ALERT {"status": 200, "problems": ["no items"], "seconds": 0.0, "bytes": 179, "items": 0} empty.xml
ALERT {"status": 200, "problems": ["stale: newest item 298 days old"], "seconds": 0.0, "bytes": 304, "items": 1, "newest_age_days": 298} stale.xml
ALERT {"status": 200, "problems": ["not well-formed: Couldn't find end of Start Tag entry lin"], "seconds": 0.0, "bytes": 300} broken.xml
ALERT {"status": 200, "problems": ["content-type 'text/html'", "no items"], "seconds": 0.0, "bytes": 37, "items": 0} html.xml
ALERT {"status": 500, "problems": ["http 500"], "seconds": 0.0} err500.xml
ALERT {"status": null, "problems": ["fetch failed: TimeoutError"], "seconds": 2.0} slow.xml
ALERT {"status": 404, "problems": ["http 404"], "seconds": 1.05} nothere.xml

Seven failures, seven different messages. Notice what a status-only check would have said: four of the seven (empty, stale, truncated, HTML) returned 200 and would have passed. The empty feed is valid XML with no entries. The stale feed is valid and its newest item is 298 days old. The truncated feed was cut off at 300 bytes mid-tag. The maintenance page came back as text/html with the text "Maintenance". The status code told you nothing in any of them.

The timeout is reported as fetch failed: TimeoutError after 2.0 seconds, which is the timeout itself; without one, a hung server would hold your job forever. (The last route's 1.05 seconds is not the server's time: my single-threaded test server was still finishing the slow request, so the 404 waited for it. In production, check from a client with a timeout and do not read much into one slow sample.)

Making the alerts useful

A health check is only as good as the message that reaches a person.

Say what and where. Include the URL, the check that failed and the observed value: "https://example.test/feed.xml: stale: newest item 298 days old (limit 14)". A message that says "feed check failed" requires opening the script.

One alert per feed per problem, not one per run. A feed that is down for a day should alert once and again when it recovers, not 24 times. Keep the last state per feed and alert on transitions.

Threshold per feed. A blog that posts weekly has a limit of 14 days; a price feed that updates hourly, 3 hours. Put thresholds in a table, not in code.

Distinguish transient from persistent. A single timeout is common. Retry once after a short wait, or require two consecutive failures before alerting.

Record the numbers. Item counts, newest age, response time and size per run, in a table. A feed whose count drops from 5,000 to 4,200 overnight passes every check above and is worth a look. Alert on a change beyond a band, such as a 20 percent drop in items.

What each check would have missed

It is worth listing which check catches which failure, because the list shows why none can be dropped. The timeout catches a server that accepts the connection and never answers. The status check catches errors and missing pages. The content-type check catches the maintenance page and the login redirect, both of which return 200 with an HTML body. The well-formedness check catches a truncated transfer and a template that emitted a bad character. The item count catches the empty feed, which a build job produces when its query returns nothing. The freshness check catches the publisher that is up and no longer updating. Each of the eight scenarios maps to exactly one check, and a feed that fails none of them is at least alive, well-formed and recent, which is what a downstream import needs before it spends effort.

Running it daily

A scheduler (cron, a systemd timer, a CI schedule, a cloud scheduler) runs the script, and the script exits non-zero when anything is in alert, which many schedulers translate into a notification. Run it from somewhere that has the same network path as your importer; a check from a laptop may pass when the server's route is blocked. Stagger it from your import job, and keep the script and its thresholds in version control, with the feed list.

Use conditional requests when you can (incremental XML feed imports by last modified). A health check that sends If-None-Match and gets 304 has confirmed the server is up and the content unchanged, with almost no transfer. It still needs the stored freshness figure from the last full fetch.

Related reading

For the overview see working with XML feeds and APIs; for the import side, parsing an Atom feed into a database; for what to do with a broken feed, dealing with XML feeds that break their own schema. The status codes are defined in RFC 9110, section 15, and the Python timeout behavior in the urllib.request documentation. Test a path in the XPath tester.

When monitoring needs more than a script

Many feeds, many owners. A monitoring system with history, dashboards, on-call routing and silencing beats a script when a team of people needs the alerts.

You need to know a feed's content is right. Health checks look at shape and freshness. Content checks (price within a range, no duplicate ids, counts in a band) belong in the import, with a reject report.

The feed is behind authentication or a CDN. The check needs the credentials and the right edge, which is a larger design.

You need an uptime record. Use an external synthetic monitoring service for the public path and keep the script for the content checks.