Analyzing Publication Metadata From XML Archives in Python

Open-access publishers, preprint servers and libraries distribute their metadata as archives of XML files, usually in a vocabulary like JATS (Journal Article Tag Suite). A bibliometrics question, how many papers per year, how many authors per paper, which keywords are rising, is a group-by over those files. The data is large in total and tiny per file, and the whole analysis fits in a short script. This post builds one, reading straight from the compressed archive, and lists what real archives do that a tidy sample hides.

The file shape

A JATS article has a front section with journal-meta and article-meta. The parts that matter here:

<article>
  <front>
    <journal-meta><journal-title>Open Parsing Letters</journal-title></journal-meta>
    <article-meta>
      <article-id pub-id-type="doi">10.9999/a00001</article-id>
      <title-group><article-title>Title 1</article-title></title-group>
      <contrib-group>
        <contrib contrib-type="author"><name><surname>S12</surname><given-names>G3</given-names></name></contrib>
      </contrib-group>
      <pub-date><year>2023</year></pub-date>
      <kwd-group><kwd>xml</kwd><kwd>schema</kwd></kwd-group>
    </article-meta>
  </front>
  <body>...</body>
</article>

Real JATS is richer: several pub-date elements with different pub-type values, affiliations linked by xref, multiple kwd-group blocks. The method is the same.

Read the archive in place

Do not unpack an archive of tens of thousands of small files to disk just to read them once. tarfile yields members one at a time, and lxml can parse the file object:

import tarfile
from collections import Counter
from lxml import etree

years, journals, kwc, authors = Counter(), Counter(), Counter(), Counter()
n = 0
with tarfile.open("pubs.tar.gz") as tar:
    for m in tar:
        r = etree.parse(tar.extractfile(m)).getroot()
        meta = r.find("front/article-meta")
        years[meta.findtext("pub-date/year")] += 1
        journals[r.findtext("front/journal-meta/journal-title")] += 1
        kwc.update(k.text for k in meta.iterfind("kwd-group/kwd"))
        authors[len(meta.findall("contrib-group/contrib[@contrib-type='author']"))] += 1
        n += 1
print(n, dict(sorted(years.items())), dict(sorted(authors.items())))

The output:

3000 articles in 0.17s
per year {'2019': 442, '2020': 405, '2021': 401, '2022': 449, '2023': 425, '2024': 429, '2025': 449}
authors per article {1: 384, 2: 795, 3: 727, 4: 357, 5: 367, 8: 370}
top keywords [('storage', 937), ('xml', 921), ('query', 913)]
journals {'Applied Schemas': 761, 'Journal of Tables': 755, 'Open Parsing Letters': 747, 'Data Systems Review': 737}

Three thousand articles in 0.17 seconds. A tar.gz is read sequentially, so the cost is the decompression plus the parse, and it scales linearly: a million articles take a few minutes on one core. Parse errors are the realistic exception. Wrap the parse in try/except etree.XMLSyntaxError, count the failures and keep going, as in summing values across thousands of XML invoices.

One predicate in that script is worth noticing: contrib[@contrib-type='author']. JATS uses the contrib element for authors, editors, translators and others, told apart by the attribute. Counting contrib alone would inflate the author counts with every editor. The same trap exists in other vocabularies: the element name is generic and an attribute carries the role.

What the counts say, and what they do not

On the generated data, the yearly counts are flat, between 401 and 449, which is what a uniform random generator produces. Authors per article peak at two and three. With a real archive, patterns would appear: growth over time, a long tail of papers with dozens of authors, seasonal bumps. The method gives you the table, and interpretation is yours.

Be careful with a few quantities that look simple.

"Year". JATS articles carry several dates: when it was received, accepted, published electronically, published in print. They have different pub-type or date-type attributes and can fall in different years. Pick one deliberately, state which, and be consistent. A count per year that mixes them is a count of nothing in particular.

"Number of authors". Authors may be listed as a group (collab) rather than as individuals, and large collaborations list thousands. Report the median and the maximum, not only the mean.

"Keywords". Authors write free text. XML, xml and Xml are three counter keys, and data-wrangling and data wrangling two more. Lowercase, strip and decide about hyphens before counting, and keep a sample of the raw forms. Controlled vocabularies, such as MeSH terms in biomedical archives, are cleaner but have their own hierarchy.

"Journal". Titles change, abbreviations vary and the same journal appears under several names. A stable identifier, an ISSN, is better than the title if the archive carries one.

Missing fields and duplicates

A real archive is not uniform. Some articles have no year, no DOI, no keywords. With findtext returning None, the Counter silently gets a None key, which is useful: it shows how many records lack the field. Print the None counts, and decide for each whether it is an error, an acceptable absence or a reason to look at the source.

Duplicates are common when archives are cumulative. The same DOI may appear in two files, a correction or a later version. Collect DOIs while streaming and count how many appear more than once. For a million DOIs, a Python set of strings is a few hundred megabytes; use a database or a sorted file if that is too much (streaming XML into PostgreSQL without a DOM is the route for bigger sets).

Finally, keep the unit of analysis clear. Counting files is not counting articles if an archive carries several versions, and counting authors is not counting people if names are not disambiguated. Author disambiguation is a research problem in itself; identifiers such as ORCID, when present in the XML as contrib-id, are the reliable key.

Writing the result down

An analysis of an archive is only as repeatable as its record of the archive. Note the file name, the size and the SHA-256 of the archive you read, the date you read it, the version of the script and the choices you made: which date counts as the year, which contributor role counts as an author, how keywords were normalized. Save the Counters as a small CSV beside the script (year,count) so a chart or a later comparison reads a file rather than rerunning the parse.

If you will repeat the analysis monthly against a growing archive, store the extracted fields once, one row per article with DOI, year, journal, author count and a list of keywords, and append new rows as new files arrive. The table is a few megabytes for a million articles, queries on it take milliseconds, and the XML parse happens once per file instead of once per question. The per-article table is also the right place to put the quality flags from this post: missing year, duplicate DOI, parse failure. Keep them as columns, and filter on them in queries, so the cleaning decisions stay visible.

Related reading

This post is part of answering questions from XML data. The publisher side of the format is in reading publisher metadata in JATS XML, and the XPath behind the predicates in XPath predicates and position. JATS is maintained as the NISO Z39.96 standard with documentation at the JATS website, and tarfile is described in the Python documentation. Test the paths in the XPath tester.

When archive metadata analysis needs different tooling

The archive is huge. At hundreds of gigabytes, a one-core Python loop takes days. Split by member ranges and run workers in parallel, or load into a warehouse with Parquet (choosing Parquet after you parse XML).

You need citation links. References are a graph. Extract them into an edge table (citing DOI, cited DOI) and use a graph query or SQL joins.

You need full text. Body text with mixed content needs text extraction, not field counting (mixed content in XML and why it hurts parsing).

The analysis is a study. For a published result, document the archive version, the selection rules and the cleaning steps. The script is the method section.