Editor's pick
CSVGuide
Handling CSV Files Too Big to Open in Excel or Python
A decision guide for CSV files too big to open: look, count, sample, query in place, chunk, split, compress or convert, with numbers from a 1.3 GB file.
17 min read
Topic
Performance
Why the query is slow, and what to change about it. Execution plans, statistics, caching and rewrites, with the numbers to prove it.
PostgreSQL
Partitioning Mistakes That Make PostgreSQL Queries Slower
Partitioning mistakes that make PostgreSQL queries slower: wrong key, too many partitions, id lookups, missing pruning, tiny partitions and fixes for each.
7 min read
JSONGuide
Handling JSON Files Too Large to Parse With Ordinary Tools
A tested guide to JSON files too big to load: memory and string limits, array to JSON Lines, DuckDB, streaming, splitting, compression and databases.
18 min read
PostgreSQL
Partition Pruning in PostgreSQL and Checking It Works
Partition pruning in PostgreSQL explained: plan time and run time pruning, what defeats it, how to read EXPLAIN output and a checklist for fixing queries.
7 min read
PostgreSQL
Indexes and Primary Keys on Partitioned Tables in PostgreSQL
Indexes and primary keys on partitioned tables in PostgreSQL: the partition key rule, local indexes, ON ONLY builds, foreign keys and lookup patterns.
7 min read
CSV
How Much Memory pandas Needs to Load a CSV File in 2026
pandas needed 3.9 GB of peak memory to load a 1.3 GB CSV, and 0.18 GB to keep two typed columns. Measure it yourself with memory_usage and a rule of thumb.
6 min read
CSV
Compressing CSV Files and Reading Them Without Unzipping
Compress a CSV with gzip or zstd and read it directly with zcat, pandas, DuckDB or Python: 130 MB became 34 to 43 MB and no unzipped copy was needed.
6 min read
CSV
Sampling Random Rows From a Huge CSV Without Loading It
Take a random sample of a 1.3 GB CSV with shuf, awk, DuckDB or a seek: all run in seconds, but line-based samples break rows that contain a newline.
6 min read
CSV
Counting Lines in a Multi Gigabyte CSV Quickly and Correctly
wc -l counts a 1.3 GB CSV in under a second and is 9,856 rows off, because quoted newlines add lines. DuckDB counts records in 1 s; csv.reader takes 15 s.
6 min read
CSV
Reading a Large CSV in Chunks With pandas Without Surprises
Read a 20-million-row CSV in chunks with pandas: 150 MB peak instead of 3.9 GB, the same answer as DuckDB, and the traps: zip as int64, mixed types, pyarrow.
9 min read
CSV
Splitting a Large CSV Into Smaller Files Without Losing Rows
split -l cut a 1.3 GB CSV into 21 parts in 3.6 s with the header in one. awk, qsv, Miller and DuckDB repeat it; a cut inside a quoted newline breaks a part.
10 min read
XML
Parsing a Large Wikipedia Style XML Dump From bz2 Directly
Stream a compressed MediaWiki style export with iterparse, skip redirects, count links and text, and keep memory near 24 MB while reading 176 MB of XML.
6 min read
Get new articles by email
We send one email when a new article is published. No spam.