XML
How Much Memory a DOM Parser Needs for a Large XML File
Measured peak memory of ElementTree and lxml on 32 MB and 127 MB order files shows 11 to 15 times the file size, and how streaming cut it to 22 MB.
6 min read
Tag
benchmark
21 posts
XMLGuide
Parsing Big XML Files Fast Without Running Out of Memory
A measured guide to large XML files: memory multipliers, streaming, library choice, COPY into PostgreSQL, parallel splits, compression and indexes.
17 min read
Excel
How Many Rows Can Pandas Read From Excel Before It Struggles
Estimate how many rows pandas can read from an Excel file for your memory, why read_excel loads everything, and how usecols and dtype shrink the load.
7 min read
Excel
Benchmarking Excel Readers in Python With a Fair Test
Compare openpyxl, calamine, pandas engines and DuckDB for reading xlsx: build test files, time runs, record peak memory and avoid common benchmark mistakes.
7 min read
Excel
Reducing Workbook Size by Removing Unused Styles and Data
Find why an xlsx is large by looking inside the zip, then remove unused styles, empty formatted rows, pivot caches, images and links to shrink the file.
8 min read
Excel
Reading a 500 MB Excel File Without Running Out of Memory
Read a very large xlsx in Python without exhausting memory: why xlsx expands, openpyxl read only mode, calamine and DuckDB, and how to measure peak memory.
7 min read
XML
Limiting XML Size and Depth for Uploads With Measured Limits
Where lxml and ElementTree stop on deep nesting, long text nodes and wide documents, what a tree costs in memory, and the limits to set for uploads.
7 min read
CSV
Removing Null Bytes and Control Characters From a CSV
A NUL byte can cut a PostgreSQL load short while Python keeps it. Find them with grep, strip them from 130 MB in under a second, and keep tabs and newlines.
6 min read
CSV
Why a CSV File Has No Types and What That Costs You
A CSV stores text, so every reader infers types. Measure what inference costs in time and memory on 2 million rows, and what it breaks: ZIP codes and dates.
8 min read