CSV
Tailing and Following a CSV That Is Still Being Written
Follow a growing CSV with tail -f, skip the header, survive rotation with tail -F and read only complete lines, plus a Python follower and the buffering trap.
6 min read
Tag
large-files
34 posts
CSV
How Much Memory pandas Needs to Load a CSV File in 2026
pandas needed 3.9 GB of peak memory to load a 1.3 GB CSV, and 0.18 GB to keep two typed columns. Measure it yourself with memory_usage and a rule of thumb.
6 min read
CSV
Compressing CSV Files and Reading Them Without Unzipping
Compress a CSV with gzip or zstd and read it directly with zcat, pandas, DuckDB or Python: 130 MB became 34 to 43 MB and no unzipped copy was needed.
6 min read
CSV
Sampling Random Rows From a Huge CSV Without Loading It
Take a random sample of a 1.3 GB CSV with shuf, awk, DuckDB or a seek: all run in seconds, but line-based samples break rows that contain a newline.
6 min read
CSV
Counting Lines in a Multi Gigabyte CSV Quickly and Correctly
wc -l counts a 1.3 GB CSV in under a second and is 9,856 rows off, because quoted newlines add lines. DuckDB counts records in 1 s; csv.reader takes 15 s.
6 min read
CSV
Reading a Large CSV in Chunks With pandas Without Surprises
Read a 20-million-row CSV in chunks with pandas: 150 MB peak instead of 3.9 GB, the same answer as DuckDB, and the traps: zip as int64, mixed types, pyarrow.
9 min read
CSV
Splitting a Large CSV Into Smaller Files Without Losing Rows
split -l cut a 1.3 GB CSV into 21 parts in 3.6 s with the header in one. awk, qsv, Miller and DuckDB repeat it; a cut inside a quoted newline breaks a part.
10 min read
XML
Parsing a Large Wikipedia Style XML Dump From bz2 Directly
Stream a compressed MediaWiki style export with iterparse, skip redirects, count links and text, and keep memory near 24 MB while reading 176 MB of XML.
6 min read
CSV
Opening a CSV File That Is Too Large for Excel or Sheets
A worksheet holds 1,048,576 rows. When a CSV has more, filter, aggregate or split it first with DuckDB, split or csvkit, and hand Excel a file that fits.
6 min read
XML
How Much Memory a DOM Parser Needs for a Large XML File
Measured peak memory of ElementTree and lxml on 32 MB and 127 MB order files shows 11 to 15 times the file size, and how streaming cut it to 22 MB.
6 min read
XMLGuide
Parsing Big XML Files Fast Without Running Out of Memory
A measured guide to large XML files: memory multipliers, streaming, library choice, COPY into PostgreSQL, parallel splits, compression and indexes.
17 min read
Excel
How Many Rows Can Pandas Read From Excel Before It Struggles
Estimate how many rows pandas can read from an Excel file for your memory, why read_excel loads everything, and how usecols and dtype shrink the load.
7 min read