Parsing an Atom Feed Into a Database With Safe Upserts

An Atom feed is a list of entries with stable identifiers and change times, which makes it one of the easiest XML formats to keep in sync with a table. The design is short: the entry id is the key, updated says when it last changed, and a conditional request tells you when nothing did. The details that go wrong are in the links, the content types and the dates.

The table and the key

create table feed_item(
  id text primary key,
  title text,
  updated timestamptz,
  first_seen timestamptz default now(),
  last_changed timestamptz default now()
);
create table feed_state(url text primary key, etag text, last_modified text, fetched_at timestamptz);

Atom requires an id on every entry. It is a URI (often a tag: URI or a permalink) that must not change for the life of the entry, which makes it the primary key. updated is a timestamp that should change when the entry changes in a meaningful way. The second table remembers the validators from the last fetch. Keep first_seen and last_changed as well: they are your own times, and they answer questions that the publisher's updated cannot ("when did we first see this?"), including when a publisher backdates or reuses a date.

The importer

req = urllib.request.Request(URL, headers={"User-Agent": "dh-feed-importer/1.0"})
if row:
    if row[0]: req.add_header("If-None-Match", row[0])
    if row[1]: req.add_header("If-Modified-Since", row[1])
try:
    resp = urllib.request.urlopen(req, timeout=10)
except urllib.error.HTTPError as e:
    if e.code == 304:
        print("304 Not Modified: nothing to parse"); sys.exit(0)
    raise
root = etree.fromstring(resp.read())
for e in root.iterfind("a:entry", A):
    i, t, u = e.findtext("a:id", namespaces=A), e.findtext("a:title", namespaces=A), e.findtext("a:updated", namespaces=A)
    cur.execute("select title, updated from feed_item where id = %s", (i,))
    old = cur.fetchone()
    if old is None:
        cur.execute("insert into feed_item(id, title, updated) values (%s, %s, %s)", (i, t, u)); ins += 1
    elif old[0] != t or old[1].isoformat().replace("+00:00", "Z") != u:
        cur.execute("update feed_item set title=%s, updated=%s, last_changed=now() where id=%s", (t, u, i)); upd += 1
    else:
        same += 1

The script fetches with conditional headers, and exits on a 304 without parsing. Otherwise it reads each entry, compares with the stored row and counts insert, update and unchanged. Running it four times, with the feed edited between the second and third run (one entry's title and date changed, one new entry added):

200 OK: 3 inserted, 0 updated, 0 unchanged | etag "3362aa0db9a5"
304 Not Modified: nothing to parse
200 OK: 1 inserted, 1 updated, 2 unchanged | etag "17d1db91d971"
304 Not Modified: nothing to parse

The first run loads three entries. The second sends If-None-Match and gets 304 with no body. After the edit, the feed's ETag changes, the third run downloads it and reports exactly the difference: e4 inserted, e2 updated, e1 and e3 unchanged. The fourth run is a 304 again. The mechanics of the conditional fetch are in incremental XML feed imports by last modified. Doing the comparison per entry with a select is simple and fine for hundreds of entries. For thousands, use one statement: insert ... on conflict (id) do update set title = excluded.title, updated = excluded.updated where feed_item.updated is distinct from excluded.updated, and read the count of affected rows.

Two choices in the comparison deserve thought. Comparing updated as text after normalizing (+00:00 to Z) is fragile; parse both sides to timestamps and compare those. And decide whether a changed updated with an unchanged title is an update: for most feeds yes, because the content may have changed.

Links, categories and content

A realistic Atom entry has more than a title:

<entry>
  <id>tag:example.test,2026:r42</id><title type="html">Release 4.2 &amp;amp; fixes</title>
  <link rel="alternate" href="https://example.test/r42"/>
  <link rel="enclosure" href="https://example.test/r42.tgz" length="1048576" type="application/gzip"/>
  <published>2026-10-20T09:00:00+02:00</published><updated>2026-10-27T08:00:00Z</updated>
  <category term="release"/><category term="security"/>
  <content type="html">&lt;p&gt;Fixes &lt;b&gt;two&lt;/b&gt; bugs.&lt;/p&gt;</content>
</entry>

Read each with its rules.

Links. There can be several, told apart by rel: alternate (the page for the entry, and the default when rel is missing), enclosure (an attachment), self, related. lxml read both for the first entry as {'alternate': ..., 'enclosure': ...}. Code that takes the first link it finds gets the right one only if the alternate happens to be first. The entry's permanent URL is the alternate link, and the second entry in the sample has a link with no rel, which means the same thing.

Dates. published is optional and updated is required. The first entry's published has a +02:00 offset and its updated ends in Z; both are ISO 8601 with a zone, and parse to timestamps with time zone. Do not strip the offset.

Categories are term attributes, zero or more: ['release', 'security'] for the first entry and none for the second.

Titles and content have types. type="text" (the default) is plain text. type="html" is HTML that has been XML-escaped. type="xhtml" is real XML inside a div. The title above is Release 4.2 & fixes in HTML terms, which after the XML parser is the string Release 4.2 & fixes (the XML layer decoded & to &, and the remaining & is HTML). Both lxml and feedparser returned the title in that form. A page that displays it as text shows & literally; one that treats it as HTML shows an ampersand. Store the text and its type, and decide at display time, with sanitization for html.

feedparser for the heavy lifting

feedparser reads the same file into a normalized structure:

feedparser: tag:example.test,2026:r42 | 'Release 4.2 &amp; fixes' | link https://example.test/r42 | published 2026-10-20T09:00:00+02:00 | updated 2026-10-27T08:00:00Z | tags ['release', 'security'] | content [('text/html', '<p>Fixes <b>two</b> ')] | summary <p>Fixes <b>two</b>  | enclosures [('https://example.test/r42.tgz', '1048576')]
feedparser: tag:example.test,2026:r41 | 'Release 4.1' | link https://example.test/r41 | published None | updated 2026-09-01T00:00:00Z | tags [] | content [] | summary Short summary only. | enclosures []

The library picked the alternate link, collected tags and enclosures, marked the content as text/html and, for the first entry, derived a summary from the content (the second had its own). It reports version: atom10 and bozo: False, where bozo is its flag for malformed input. For a pipeline that reads many feeds in different dialects, this uniformity is worth having, and the importer's table can be filled from entry.id, entry.title and entry.updated. The cost is that you accept its interpretation of the quirks. For a feed you control or have studied, lxml shows exactly what the file says. Both are reasonable; use feedparser to discover and lxml to load when exactness matters.

Habits

Key on id, never on link or title: titles change and permalinks get rewritten. Keep the raw entry XML in a column if disk is cheap, because the day a parser rule changes you can reprocess. Handle deletion deliberately: a feed shows a window of recent entries, and an entry that leaves the window has not been deleted. Do not delete rows the feed no longer lists. Log the counts of each run (inserted, updated, unchanged, rejected) and alert when they leave their range.

Related reading

For the conditional fetch, see incremental XML feed imports by last modified; for the formats, RSS and Atom feeds explained for developers; for the overview, working with XML feeds and APIs. Atom is specified in RFC 4287, and feedparser is documented at feedparser.readthedocs.io. Test a path in the XPath tester.

When feedparser alone is enough

You show feeds, not store them. A reader that renders the latest entries needs no table. Let feedparser normalize and display.

You do not need history. If you only ever want the current window, replace the table's contents each run.

The feeds are many and unknown. feedparser tolerates broken feeds and dialects that a strict loader would reject.

You need exact control. For a partner feed with a contract, read it strictly and reject deviations (dealing with XML feeds that break their own schema).