Incremental XML Feed Imports by Last Modified and ETag

The cheapest feed import is the one that does nothing. Feeds, sitemaps and data files change rarely compared with how often they are polled, and HTTP has had a mechanism for "has this changed?" since the nineties. Using it saves bandwidth for the publisher and parse time for you, and it is the polite way to poll. This article shows the exchange with real requests and the state you need to keep.

The two validators

A server that supports conditional requests sends validators with a response:

HTTP/1.0 200 OK
Content-Type: application/atom+xml; charset=utf-8
ETag: "3362aa0db9a5"
Last-Modified: Tue, 27 Oct 2026 14:05:48 GMT
Content-Length: 552

ETag is an opaque identifier of this version of the resource, often a hash or a version number. Last-Modified is a date. The client stores both and sends them back on the next request:

GET /atom.xml
If-None-Match: "3362aa0db9a5"
If-Modified-Since: Tue, 27 Oct 2026 14:05:48 GMT

If nothing changed, the server answers 304 Not Modified, with the validators and no body:

HTTP/1.0 304 Not Modified
ETag: "3362aa0db9a5"
Last-Modified: Tue, 27 Oct 2026 14:05:48 GMT

The client uses its stored copy, or in an import job, does nothing. If the resource changed, the server answers 200 with the new body and new validators. I ran curl with and without the header against the test server to see both.

When a client sends both headers, the rules say the ETag is evaluated first and Last-Modified is only considered when there is no ETag comparison. My test server follows that, and so should yours: store both, send both, and expect the ETag to decide. ETag is the better validator for a second reason: Last-Modified has a resolution of one second, so a change made within the same second as the stored value cannot be detected by a date comparison. An ETag derived from the content has no such blind spot.

The importer

cur.execute("select etag, last_modified from feed_state where url = %s", (URL,))
row = cur.fetchone()
req = urllib.request.Request(URL, headers={"User-Agent": "dh-feed-importer/1.0"})
if row:
    if row[0]: req.add_header("If-None-Match", row[0])
    if row[1]: req.add_header("If-Modified-Since", row[1])
try:
    resp = urllib.request.urlopen(req, timeout=10)
except urllib.error.HTTPError as e:
    if e.code == 304:
        print("304 Not Modified: nothing to parse"); sys.exit(0)
    raise
...
cur.execute("insert into feed_state values (%s, %s, %s, now()) on conflict (url) do update set etag=excluded.etag, last_modified=excluded.last_modified, fetched_at=now()",
            (URL, resp.headers.get("ETag"), resp.headers.get("Last-Modified")))

Note a detail of urllib: a 304 is raised as an HTTPError, because the library treats any non-2xx response as an error. requests returns it as a normal response with status_code == 304. Handle whichever your client does, and make a 304 an expected outcome, not a failure: it should exit with success and log "unchanged". The state is stored after the data is loaded successfully. If the load fails halfway, the old validators remain, and the next run fetches again rather than skipping content you never saved.

Running it against the server four times, with the feed edited between the second and third:

200 OK: 3 inserted, 0 updated, 0 unchanged | etag "3362aa0db9a5"
304 Not Modified: nothing to parse
200 OK: 1 inserted, 1 updated, 2 unchanged | etag "17d1db91d971"
304 Not Modified: nothing to parse

Half of the requests transferred no feed at all. The third shows the second layer of incrementality: when the feed does change, the importer compares per entry on id and updated, so only the new and changed entries touch the database (parsing an Atom feed into a database). HTTP validators save the transfer and the parse; the per-entry comparison saves the writes.

When the server ignores validators

Many servers, especially for dynamically generated feeds, send no ETag or ignore conditional headers and answer 200 with the whole body every time. You can still avoid the work downstream by hashing the body:

digest = hashlib.sha256(body).hexdigest()
if digest == stored_digest: return   # identical content, skip parse and load

This does not save the download, which is the courtesy to the publisher, but it saves the parse and the database round trips. Be careful: feeds that embed a generation timestamp or a random token produce a different hash each time, and the hash never matches. Hash the canonical form of the entries in that case, or the sorted list of (id, updated) pairs, which captures what matters.

Also respect Cache-Control: max-age and Expires, which tell you not to ask at all for a while, and Retry-After on 429 and 503 responses. A poll interval should be at least as long as the feed's typical change interval; polling a feed that updates daily every minute is the behavior that gets clients blocked.

Other incremental mechanisms

HTTP validators are the generic mechanism. Many feeds offer better ones, and they are worth using when they exist.

A since parameter. ?updated_since=2026-10-26T00:00:00Z or ?since=<token> returns only what changed. Store the newest updated you saw (or the server's returned token) and ask for everything after it, with a small overlap to tolerate clock differences, and de-duplicate by id.

A delta feed. Some publishers produce daily change files (changes-2026-10-26.xml) alongside a full snapshot. Apply the deltas in order and re-sync from the snapshot on a schedule.

A manifest. A sitemap index lists files with their lastmod; refetch only files whose date advanced (reading a sitemap XML file).

WebSub or webhooks, where the publisher pushes a notification, remove polling altogether.

For each, the importer has the same shape: a stored cursor, a fetch that depends on it, and an idempotent load keyed on a stable id.

Failure modes

State saved before the load succeeded. Store validators after success.

A 304 treated as an error. It is the common case in steady state.

A lost state table. The next run fetches everything, which the idempotent load tolerates. That is why the load must be idempotent.

Clock drift in Last-Modified comparisons. Use the server's value as given; do not generate your own.

Proxies and CDNs that strip or rewrite validators. If a feed behind a CDN never returns 304, check the response headers directly with curl -I.

Related reading

For the table side see parsing an Atom feed into a database; for downloads that arrive compressed, handling compressed XML downloads; for the overview, working with XML feeds and APIs. Conditional requests are specified in RFC 9110, section 13, and caching in RFC 9111. Test a path in the XPath tester.

When the feed offers a better incremental mechanism

There is a since parameter or a delta feed. Use it. It scales with the number of changes, not with the size of the feed.

The feed is small and cheap. A 5 KB feed fetched every ten minutes costs little, and the added state may not pay for itself.

You need every change, not the latest state. Polling sees the state at poll time; changes between polls collapse. For audit trails, use a source with an event log.

The publisher asks you not to poll. Subscribe to their notifications.