CSV
Reading a Large CSV in Chunks With pandas Without Surprises
Read a 20-million-row CSV in chunks with pandas: 150 MB peak instead of 3.9 GB, the same answer as DuckDB, and the traps: zip as int64, mixed types, pyarrow.
9 min read
Topic
CSV
The format everyone exports and nobody specifies. Delimiters, quoting, encodings and loading it into a database intact.
CSV
Splitting a Large CSV Into Smaller Files Without Losing Rows
split -l cut a 1.3 GB CSV into 21 parts in 3.6 s with the header in one. awk, qsv, Miller and DuckDB repeat it; a cut inside a quoted newline breaks a part.
10 min read
CSV
Opening a CSV File That Is Too Large for Excel or Sheets
A worksheet holds 1,048,576 rows. When a CSV has more, filter, aggregate or split it first with DuckDB, split or csvkit, and hand Excel a file that fits.
6 min read
CSV
Converting CSV to Markdown Tables for Documentation
pandas, Miller and csvlook turn a CSV into a Markdown table, and each breaks on a pipe or a newline. pandas wrote 12.50 as 12.5 and csvlook 02134 as 2,134.
9 min read
CSV
Converting CSV to SQL INSERT Statements the Safe Way
Generate INSERT statements from a CSV without breaking on O'Neil or running a DROP TABLE hidden in a value. Batches of 1,000 loaded 193,708 rows in 1.6 s.
13 min read
CSV
Converting JSON to CSV When the Data Is Nested With Arrays
Flatten nested JSON to CSV with pandas, Python or DuckDB: dotted columns, a row per array item, the order that vanishes and arrays written as Python text.
10 min read
CSV
Converting Excel to CSV Without Losing Leading Zeros
pandas turned a text cell 02134 into 2134 and in2csv did too; qsv kept it. A number formatted 00000 loses its zeros everywhere unless you rebuild them.
10 min read
CSV
Converting CSV to an Excel Workbook With Formatting
Write a CSV to .xlsx with xlsxwriter: text zips, number formats, real dates and a frozen header. write() made =1+1 a formula; rows past 1,048,575 vanished.
8 min read
CSV
Converting CSV to YAML for Config Seeds Without Surprises
Hand-written YAML turned NO into false, 02134 into 1116 and 12:30 into 750; yaml.safe_dump quotes them. Dumping 193,708 records took 53 s, 15 s with libyaml.
9 min read
CSV
Converting CSV to XML With a Short and Safe Python Script
Convert CSV to XML with the standard library: invalid tag names and control characters need your own checks; a streaming writer does 193,708 rows in 22 MB.
10 min read
CSV
Converting CSV to JSON Lines for Streaming Pipelines
Convert CSV to JSON Lines, one object per line: a 13 MB streaming script, DuckDB in 0.3 s, files that split cleanly and a cut-off file that shows itself.
9 min read
CSV
Converting CSV to JSON With Python Without Losing Types
Convert CSV to JSON with the csv module, then decide on types: zip stays text, money stays exact. pandas gave 2134.0 for 02134; converters took 0.4 to 36 s.
9 min read
Get new articles by email
We send one email when a new article is published. No spam.