CSV
Using git diff Sensibly on a CSV File With Word Diff
Make git diff readable for a CSV: word diff on commas, a textconv that aligns columns, a sorted file with one record per line and a trailing newline.
6 min read
Topic
Data Wrangling
Take messy CSV, JSON and spreadsheets and land them in tables without losing a row or a character on the way.
CSV
Calculating Month Over Month Growth From a CSV in SQL
Compute month over month growth with lag(): revenue of 46.50, 202.00 and 382.00 grew 334.4% and then 89.1%. Partial months, gaps and a zero base need care.
6 min read
CSV
Filling Gaps in a CSV Time Series With SQL and DuckDB
Find and fill missing days in a CSV time series: build a calendar with range(), left join the data and choose zero, carry-forward or interpolation.
7 min read
CSV
Joining Two CSV Files on a Messy Key Without Losing Rows
Join two CSVs on a messy key: a strict join matched 1 of 4 orders, a normalized key matched 3, and an anti-join listed the rest. Check counts and duplicates.
6 min read
CSV
Tailing and Following a CSV That Is Still Being Written
Follow a growing CSV with tail -f, skip the header, survive rotation with tail -F and read only complete lines, plus a Python follower and the buffering trap.
6 min read
CSV
How Much Memory pandas Needs to Load a CSV File in 2026
pandas needed 3.9 GB of peak memory to load a 1.3 GB CSV, and 0.18 GB to keep two typed columns. Measure it yourself with memory_usage and a rule of thumb.
6 min read
CSV
Compressing CSV Files and Reading Them Without Unzipping
Compress a CSV with gzip or zstd and read it directly with zcat, pandas, DuckDB or Python: 130 MB became 34 to 43 MB and no unzipped copy was needed.
6 min read
CSV
Sampling Random Rows From a Huge CSV Without Loading It
Take a random sample of a 1.3 GB CSV with shuf, awk, DuckDB or a seek: all run in seconds, but line-based samples break rows that contain a newline.
6 min read
CSV
Counting Lines in a Multi Gigabyte CSV Quickly and Correctly
wc -l counts a 1.3 GB CSV in under a second and is 9,856 rows off, because quoted newlines add lines. DuckDB counts records in 1 s; csv.reader takes 15 s.
6 min read
CSV
Reading a Large CSV in Chunks With pandas Without Surprises
Read a 20-million-row CSV in chunks with pandas: 150 MB peak instead of 3.9 GB, the same answer as DuckDB, and the traps: zip as int64, mixed types, pyarrow.
9 min read
CSV
Splitting a Large CSV Into Smaller Files Without Losing Rows
split -l cut a 1.3 GB CSV into 21 parts in 3.6 s with the header in one. awk, qsv, Miller and DuckDB repeat it; a cut inside a quoted newline breaks a part.
10 min read
CSV
Opening a CSV File That Is Too Large for Excel or Sheets
A worksheet holds 1,048,576 rows. When a CSV has more, filter, aggregate or split it first with DuckDB, split or csvkit, and hand Excel a file that fits.
6 min read
Get new articles by email
We send one email when a new article is published. No spam.