Extracting Time Series From XML Sensor Files With Pandas

Sensors, meters, loggers and trackers write XML more often than people expect. The format is verbose and the readings are tiny, but it is what the device vendor chose. The analysis you want is simple: a time series per sensor, hourly or daily summaries, a plot. Getting there takes four steps, and two of them are not about parsing: checking for gaps and checking for jumps. Those checks are what make the numbers believable. The example below has a deliberate fault of each kind.

The file

<sensors>
  <sensor id="s1" unit="C">
    <reading t="2026-03-01T00:00:00Z" v="20.01"/>
    <reading t="2026-03-01T00:00:10Z" v="19.98"/>
    ...

Each sensor has an id and a unit; each reading has a timestamp t and a value v as attributes. The generated file is 1,153,021 bytes. It is a common layout: repeated small elements, attributes for the data, a unit declared once at the top.

Step 1: into a table

Walk the sensors and readings and build rows. For files of this size a list of tuples is fine; for bigger ones stream with iterparse (parsing large XML files with iterparse):

import pandas as pd
from lxml import etree

root = etree.parse("sensors.xml").getroot()
rows = [(s.get("id"), r.get("t"), float(r.get("v"))) for s in root for r in s]
df = pd.DataFrame(rows, columns=["sensor", "t", "v"])
df["t"] = pd.to_datetime(df["t"], utc=True)
print(df.groupby("sensor").size().to_dict())

{'s1': 8640, 's2': 8340, 's3': 8640}

A day at one reading per ten seconds is 8,640 readings. Sensors one and three have exactly that; sensor two has 8,340, which is 300 short. That is the first finding, available from one line of output. The timestamp conversion deserves care: utc=True interprets the trailing Z and gives a timezone-aware column, so later comparisons and resampling are not at the mercy of the machine's local zone. If the file carries local times without an offset, find out the zone from the vendor, and do not guess.

Step 2: check for gaps

Before averaging anything, look at the spacing of the timestamps within each sensor:

d = df.sort_values(["sensor", "t"]).assign(dt=lambda d: d.groupby("sensor")["t"].diff())
print(d[d["dt"] > pd.Timedelta(minutes=1)][["sensor", "t", "dt"]].to_string(index=False))

sensor                        t              dt
    s2 2026-03-01 09:10:00+00:00 0 days 00:50:10

Sensor two resumed at 09:10:00 after 50 minutes and ten seconds without data. The threshold of one minute is six times the normal cadence; choose one from the device's documented interval. A gap is a fact about the data, and it matters for the summaries: an hourly mean over the hour that contains the gap is computed from fewer readings than its neighbors. Decide what to do and record it. You can leave the hole in the series, interpolate across short gaps (and say so), or flag hours with fewer readings than a minimum. Do not let a resample fill the hole silently.

Step 3: check for jumps

Sudden changes in the value are the other class of problem. Compute the absolute difference between consecutive readings of each sensor and look at the large ones:

j = df.sort_values(["sensor", "t"]).assign(jump=lambda d: d.groupby("sensor")["v"].diff().abs())
print(j[j["jump"] > 5][["sensor", "t", "v", "jump"]].to_string(index=False))

sensor                         t    v  jump
    s3 2026-03-01 16:40:00+00:00 73.6 50.44

One reading in 25,620 jumped by more than five degrees: sensor three, at 16:40, from about 23 to 73.6. A room does not warm by 50 degrees in ten seconds. The unit attribute on the sensor still says C, and the device began reporting Fahrenheit from that moment. The file is well-formed and the schema would pass; only the values are wrong. This is the kind of fault no parser can find and one line of arithmetic can.

The hourly means show how it would have gone unnoticed:

h = df.set_index("t").groupby("sensor")["v"].resample("1h").mean().unstack(0)
print(h.round(2).iloc[[0, 5, 10, 16, 17, 18, 23]])

sensor                        s1     s2     s3
t                                             
2026-03-01 00:00:00+00:00  20.32  20.31  20.53
2026-03-01 05:00:00+00:00  18.12  22.13  20.95
2026-03-01 10:00:00+00:00  19.78  22.14  22.42
2026-03-01 16:00:00+00:00  21.11  19.95  39.91
2026-03-01 17:00:00+00:00  21.80  19.17  74.02
2026-03-01 18:00:00+00:00  21.28  19.32  73.83
2026-03-01 23:00:00+00:00  20.52  20.68  74.24

Sensor three's mean for the 16:00 hour is 39.91, a blend of 40 minutes of Celsius and 20 of Fahrenheit, and from 17:00 on it sits near 74. Every number is plausible as a number and the series is garbage. A dashboard that plotted this would show a dramatic heat event. The jump check finds the exact second. Once you know, you can convert the readings after that point ((v - 32) * 5 / 9), or exclude them and ask the vendor. The right fix depends on how sure you are that the change is a unit change and not a fault, which is another reason to ask.

Step 4: summarize, with the quality flags

After the checks, resample as planned and carry the quality information along. Count readings per bucket as well as the mean, so a low count is visible:

g = df.set_index("t").groupby("sensor")["v"].resample("1h").agg(["mean", "count"])

Hours with fewer than the expected 360 readings (six per minute for an hour) are suspect, and the hour containing the 50 minute gap shows it immediately. Publish the flags with the numbers. A mean of 19.17 with a count of 60 is honest, and the same mean with no count looks like any other hour.

For long histories, convert once to Parquet, partitioned by day, and keep the XML as the raw layer (storing raw XML in a lake and parsing it later). For time series with high rates, databases built for it (TimescaleDB, ClickHouse) are the better home than a DataFrame.

Recording what you decided

The checks produce decisions, and decisions belong in the output, not in your head. For each sensor, keep a small table of findings: first and last timestamp, number of readings, expected number, list of gaps with start and length, list of jumps with time and size, and the action taken for each (kept, interpolated, converted, excluded). Save it beside the series. When someone asks why the hourly mean for sensor three at 16:00 differs from the vendor's dashboard, the answer is a row in a table.

Treat the unit attribute as a claim to verify and not as a fact. Here the file said Celsius for the whole day and the data said otherwise for the last seven hours. A range check per sensor type, such as indoor air temperature between minus ten and fifty degrees, catches most of these cases in one line, and it catches them even when the jump happens slowly, which a difference check can miss. Combine both: a plausible range and a maximum step. Neither costs anything to run and both have rescued reports that looked finished.

Related reading

This post is part of answering questions from XML data. The parsing options are in converting XML to a pandas DataFrame, and the memory view in how much memory a DOM parser needs. Pandas documents resampling and the to_datetime function, and ISO 8601 timestamps are described in RFC 3339. Reformat a sample in the XML formatter.

When pandas is not the right tool for sensor XML

The data is large. A year of one-second readings from hundreds of sensors is billions of rows. Stream into a time-series database or Parquet and query there.

You need real-time alerts. A batch DataFrame answers yesterday's question. Alerts on a stream need a streaming process with the same gap and jump rules applied per reading.

The format is a standard with its own tools. Some domains (GPX tracks, meteorological XML, SensorML) have libraries that understand the vocabulary and its units. Use them where they exist.

Values need calibration. Raw counts with a per-device calibration curve are a different job from clean degrees. Keep the raw values and apply the curve as a documented step.