Editor's pick
CSVGuide
Handling CSV Files Too Big to Open in Excel or Python
A decision guide for CSV files too big to open: look, count, sample, query in place, chunk, split, compress or convert, with numbers from a 1.3 GB file.
17 min read
Tag
large-files
34 posts
XML
Compressing XML Archives and Reading Them In Place
A 127 MB XML file shrank to 14.8 MB with gzip, 15.4 MB with zstd and 10.8 MB with xz. Costs in seconds, and how to parse a gzip file without unpacking.
6 min read
XML
Parallel Processing of Split XML Files With Python Pool
Split a 127 MB XML file into four parts in 3.4 seconds, then process them in parallel for a 3.8 times speedup. Code, results and the limits.
6 min read
CSV
Sampling a CSV for a Fast First Look at Large Data
Sample a large CSV for a quick first look: why the top rows mislead, random and reproducible sampling in DuckDB, and how the error shrinks with sample size.
6 min read
XML
Handling Compressed XML Downloads Without Surprises
A gzip file, gzip transfer encoding and unlabeled gzip behave differently in requests and urllib. Sniff the bytes, stream 36.6 MB at 21 MB, and guard bombs.
6 min read
CSV
Polars for Large CSV Processing Beyond Memory Limits
Use Polars lazy scanning to filter and group a large CSV without loading it whole, set column types so zip codes stay text, and write Parquet as you go.
6 min read
XML
Streaming Large XML Files in Java Without Memory Trouble
Measured heap needs for 36.6 MB of XML in Java: DOM fails at 256 MB, StAX with JAXB per record runs in 16 MB, and the memory is flat whatever the file size.
6 min read
CSV
Limiting File Size and Row Count for CSV Uploads Safely
Pick CSV upload limits from measured parse and load speed, enforce each where it is cheapest, and reject bad shapes with a script that names the line.
7 min read
XML
DOM Versus SAX Versus StAX Parsing in Java Compared
The three Java XML APIs on one 36.6 MB file: DOM ran out of memory at a 256 MB heap and took 1.03 s with 1 GB, while SAX and StAX ran in 32 MB in about 0.3 s.
7 min read
XML
Splitting a Big XML File by Repeating Element Safely
Split a 36.6 MB XML file into eight valid files of 25,000 orders with iterparse in 1.5 seconds at 21 MB, and see why split -l produces broken XML.
6 min read
CSV
Opening a CSV With More Than a Million Rows in Excel
Excel stops at 1,048,576 rows and drops the rest if you save. Four ways to work with a bigger CSV: aggregate it, split it, use the Data Model or sample.
6 min read
JSONGuide
Handling JSON Files Too Large to Parse With Ordinary Tools
A tested guide to JSON files too big to load: memory and string limits, array to JSON Lines, DuckDB, streaming, splitting, compression and databases.
18 min read