Parsing a Large Wikipedia Style XML Dump From bz2 Directly

The dumps that Wikipedia and its sister projects publish are the classic large XML file: tens of gigabytes compressed, hundreds uncompressed, a flat sequence of page elements. People read them to extract text, links, titles, redirects or revision histories. The right tool is a stream, and the details that make it work are small. This post builds the loop on a synthetic export in the same shape, so the numbers can be reproduced without downloading a dump, and says what changes on the real thing.

The shape of an export

An export is one mediawiki root with a siteinfo block and then one page per article, each with a title, a namespace number, an id, an optional redirect element and one or more revision children holding the text:

<mediawiki xmlns="http://www.mediawiki.org/xml/export-0.11/" version="0.11" xml:lang="en">
  <siteinfo><sitename>Toy</sitename></siteinfo>
  <page>
    <title>Page 1</title><ns>0</ns><id>1</id>
    <revision><id>10</id><timestamp>2026-01-01T00:00:00Z</timestamp>
      <text xml:space="preserve">Lorem ipsum ... [[Page 7]] {{cite web|url=x}}</text></revision>
  </page>
  <page>
    <title>Redirect 7</title><ns>0</ns><id>7</id><redirect title="Page 6"/>
    <revision><id>70</id><text>#REDIRECT [[Page 6]]</text></revision>
  </page>

Three things to keep in mind. The elements are in a namespace, with a version number in its URI; it changes between dump versions, so read it from the file you have. ns is the wiki namespace number: 0 is the main article space, and others are talk pages (1), project pages (4) and so on. And a redirect is a page with a redirect element and a one-line text, which looks like an article if you count pages.

The loop

Open the compressed file with bz2.open and hand the file object to iterparse. Filter on the page tag with its full namespace:

import bz2, re
from lxml import etree

NS = "http://www.mediawiki.org/xml/export-0.11/"
pages = redirects = links = textlen = ns0 = 0
with bz2.open("toywiki.xml.bz2", "rb") as f:
    for _, el in etree.iterparse(f, events=("end",), tag=f"{{{NS}}}page"):
        if el.find(f"{{{NS}}}redirect") is not None:
            redirects += 1
        elif el.findtext(f"{{{NS}}}ns") == "0":
            t = el.findtext(f"{{{NS}}}revision/{{{NS}}}text") or ""
            ns0 += 1
            textlen += len(t)
            links += len(re.findall(r"\[\[", t))
        pages += 1
        el.clear()
        while el.getprevious() is not None:
            del el.getparent()[0]

bz2 bytes 2921313
pages=60000 redirects=8571 articles(ns0)=30940 chars=99601665 links=464423 time=2.8s peak_rss=24MB

The file is 2.9 MB compressed and 175.7 MB when decompressed, and the loop reads it in 2.8 seconds with a peak of 24 MB of memory. The memory does not depend on the dump size: only one page is in memory at a time, because each is cleared and the earlier siblings are deleted after processing (parsing big XML files fast explains why that second step matters). The same code on a 100 GB dump uses the same 24 MB plus whatever the biggest page needs.

The namespace braces are the usual stumbling block. tag="page" matches nothing in a namespaced document, so the loop silently processes zero pages and prints zeros; the format string {{{NS}}}page builds {http://...}page. If your loop's counters come out as zero on a file you know is full, suspect this first.

Counting what you mean

The counters reveal why each rule is there. 60,000 pages, of which 8,571 are redirects, leaves 51,429 real pages, and of those 30,940 are in namespace 0. Without the redirect check, redirects (whose text is only #REDIRECT [[...]]) would add 8,571 pages and 8,571 spurious links. Without the namespace check, talk and project pages would be counted as articles. Order matters in the if: a redirect in namespace 0 must be caught before the namespace test, or it is counted as an article.

The link count uses a crude regular expression on the raw wiki text, which counts every [[, including links to files and categories ([[File:...]], [[Category:...]]). Real analysis of wikitext is a parsing problem of its own: templates, nested links, comments, <nowiki> blocks. For anything beyond rough counts, use a wikitext parser (mwparserfromhell in Python), and test it on a sample before running it on the dump.

If you keep every revision (a "history" dump rather than "current pages"), each page has many revision children, and the file is hundreds of times larger. Handle the revisions inside the page loop, and aggregate there; never collect them all.

Reading without unpacking, and faster

bz2.open decompresses on a single core inside your process. For the largest dumps, two improvements help. Run the decompressor in a separate process and pipe it in: lbzip2 -dc dump.xml.bz2 | python parse.py, with the script reading sys.stdin.buffer, uses two cores and a parallel decompressor if you have one. And split the dump: dumps are usually published in numbered parts, and you can process the parts in parallel with a pool (parallel processing of split XML files). Wikimedia also publishes an index of byte offsets in multistream dumps so that a single page can be read without scanning, which is the right tool for random access.

What changes on a real dump

Real prose compresses far less than this repeated filler, which shrank to 1.7 percent of its size, so a real dump is much bigger on disk for the same number of pages, and decompression dominates the time. The largest language editions decompress to many tens of gigabytes, and a loop of this kind takes a long time there. I did not run one. Pages are much longer and more irregular, so regular expressions cost more. Plan the run: start with the first part or a few thousand pages, time them, and extrapolate. Save the extracted rows to Parquet as you go, so a failure at 80 percent does not cost the first hour.

Related reading

This post belongs to XML problems worked from real data. The decompression options are discussed in compressing XML archives and reading them in place, and the iterparse API in parsing large XML files with iterparse. The export format is documented on MediaWiki's export page and the dump files on the Wikimedia dumps site. For wikitext, see mwparserfromhell. Inspect a page element in the XML formatter.

When a dump needs a specialized tool

You need clean text. Wikitext to plain text is not a regular-expression job. Use a parser or a tool built for it, such as a text extractor that handles templates and tables.

You need a link graph. Resolve redirects, normalize titles (case of the first letter, underscores) and keep namespaces. Build the graph from a SQL dump of the link table if one is published, which is much cheaper than extracting links from text.

You need random access. Use the multistream dump and its index, or load the pages into a database keyed by title.

You only need a few pages. Ask the API. Downloading a dump to read ten pages is the long way around.