Storing Raw XML in a Data Lake and Parsing It Later

When a feed arrives as XML, the first instinct is to parse it on the way in and store the result. It is a reasonable instinct, and it loses something you will want later: the original. The day the business changes what "revenue" means, the day you discover that a field you ignored was important, the day a provider says "that file was wrong, here is the corrected one", you need to go back to the source. A data lake with a raw layer makes that a rerun instead of a project.

The two layers

A lake for this feed has at least two layers, with different rules.

Raw. The files exactly as received. Never edited, never deleted by pipelines, compressed for storage, named by source and arrival date. Anyone can parse it again.

Parsed. Typed, flattened, query-ready tables, such as Parquet files or warehouse tables. Rebuilt from raw whenever the rules change. A parsed table is a cache of an interpretation, and like any cache it can be thrown away.

The rule that binds them: every parsed row can be traced to the raw file it came from, by file name or an id carried through the parse. When someone asks "where did this number come from?", you open the raw file and read the record.

The landing step

The landing job does little on purpose. It stores the bytes, computes a checksum, counts the records and writes a manifest next to the file:

import gzip, hashlib, json, os
from lxml import etree

def land(src, source, dt, root="/lake"):
    d = f"{root}/raw/source={source}/dt={dt}"
    os.makedirs(d, exist_ok=True)
    raw = open(src, "rb").read()
    sha = hashlib.sha256(raw).hexdigest()
    name = f"{d}/{os.path.basename(src)}.gz"
    with gzip.open(name, "wb", compresslevel=6) as f:
        f.write(raw)
    n = sum(1 for _ in etree.iterparse(src, events=("end",), tag="order"))
    manifest = {"source": source, "dt": dt, "file": os.path.basename(name),
                "bytes": len(raw), "stored_bytes": os.path.getsize(name),
                "sha256": sha, "records": n, "received": "2026-10-27T09:00:00Z"}
    json.dump(manifest, open(f"{d}/_manifest.json", "w"), indent=1)
    return manifest

Run on the orders file, it wrote this manifest:

{
 "source": "orders",
 "dt": "2026-10-27",
 "file": "o100k.xml.gz",
 "bytes": 33048205,
 "stored_bytes": 3696037,
 "sha256": "3fd6050e3d7fd799df9544f257c7890ded4edb4f8afd661877aa2b9ce80980ef",
 "records": 100000,
 "received": "2026-10-27T09:00:00Z"
}

and this layout:

raw/source=orders/dt=2026-10-27/_manifest.json
raw/source=orders/dt=2026-10-27/o100k.xml.gz

The 33 MB file is stored in 3.7 MB, 11 percent, which is the compression you get from XML's repeated names (compressing XML archives and reading them in place). The path uses key=value segments, the Hive-style partition convention that Spark, Athena, BigQuery external tables and DuckDB all understand, so a query engine can prune by source and dt without scanning the rest.

Notice what the landing step does not do. It does not validate against a schema, fix encodings, drop records or convert anything. A file that arrives broken is stored broken, and the manifest and the parse step report it. If the lander rejects bad files, you lose the evidence, and the provider's "we sent it fine" has nothing to be checked against. Quarantine is better than rejection: land everything, mark the manifest "status": "failed_parse" after the parse step, and alert.

What the manifest is for

Four fields do most of the work.

The checksum of the original bytes tells you the stored file is the file received, and detects a repeat delivery: if the same SHA-256 arrives twice, skip it. It also lets a parse job record "built from sha 3fd6..." in its output, which makes lineage a lookup.

The record count, taken at landing from a cheap streaming pass, is the first reconciliation number. After parsing, the parsed table must have at least this many orders, and a smaller count is a defect. Counting on landing, not on parsing, means a parser bug cannot hide itself.

Sizes make anomalies visible. A feed that is normally 33 MB and arrives at 3 MB is wrong before anyone opens it.

The received time, kept apart from the business date inside the file, lets you reconstruct what you knew when. That matters when a corrected file arrives a week late and the report from last Tuesday has to be explained.

Parsing later, repeatably

The parse job reads the manifest, opens the raw file, streams the records and writes parsed output to a path derived from the raw one: parsed/orders/dt=2026-10-27/part-0.parquet. Because the output location is a function of the input, the job is idempotent: running it twice overwrites the same files, not duplicates. Add the raw file's SHA-256 as a column or in the file metadata, and stale output is detectable.

Version the parser. Write parser_version in the output metadata. When you change a rule, bump it and rebuild the affected partitions from raw. A comparison of old and new outputs for one partition is the best review of a rule change you will get, because it shows exactly which numbers move.

For the parsing itself, see the streaming approaches in this cluster: streaming XML into PostgreSQL without a DOM for a database target, and choosing Parquet after you parse XML for a lake target.

Operating the raw layer

Immutability. Turn on versioning or object lock where your storage offers it, and give pipelines write access to raw/ only for new keys. An overwrite of a raw file is the one mistake that cannot be repaired from inside the lake.

Retention. Keep raw files as long as the business needs to reproduce its numbers, and longer for regulated data. Storage is cheap relative to a lost source, and archive tiers reduce the cost for old partitions.

Access. Raw files contain everything the provider sent, including fields you never use and might not be allowed to keep. Apply the same classification as to the tables built from them, and consider encrypting at rest with keys per source.

Cost. At 11 percent of the original size, a year of daily 33 MB files is about 1.3 GB. The cost is not the issue; the discipline is.

Related reading

This post is part of XML in warehouses and lakes. The incremental case, where each delivery contains only changes, is in incremental XML feed imports from the last run, and the daily health check in checking feed health with a daily script. The partition naming follows the Apache Hive partitioning convention, and SHA-256 is specified in FIPS 180-4. Check a landed file's structure in the XML formatter.

When keeping raw XML is not worth the storage

The source is reproducible. If the provider keeps every delivery and lets you refetch any date, your raw layer is a convenience, not a necessity. Keep the manifest and a retention shorter than theirs.

The payload is sensitive and you do not need it. If a feed carries personal data you are not entitled to keep, filter it at the boundary and say so in the manifest, rather than storing it raw.

The files are enormous and rarely reparsed. Terabytes of XML whose parsed form never changes cost real money. Keep the parsed layer and a sampled raw set, and write the decision down.

The feed is a stream of tiny messages. Landing each message as a file creates millions of objects. Batch them into hourly files before landing.