CSV
Reading a CSV With Inconsistent Column Counts in Python
Find the rows in a CSV whose field count differs from the header, learn why they happen, and quarantine or repair them in Python without losing data.
8 min read
Topic
Data Wrangling
Take messy CSV, JSON and spreadsheets and land them in tables without losing a row or a character on the way.
CSV
Why pandas Turns ZIP Codes Into Numbers and How to Stop It
pandas read_csv infers a column of digits as integers, so 02134 becomes 2134. See the cause, the 190,451 broken codes in 2M rows, and the fixes that work.
7 min read
CSV
Parsing Dates While Reading a CSV With pandas Correctly
Use parse_dates and date_format in pandas read_csv, then handle mixed formats and day-first files, where format="mixed" guessed two conventions in one column.
8 min read
CSV
Reading Only Some Columns of a Large CSV With pandas
Use usecols, dtype, nrows and chunksize to load a large CSV into pandas with a fraction of the memory. Measured on 2,000,000 rows, with the float32 trap.
8 min read
CSV
pandas read_csv Dtype Warnings and How to Fix Them
What DtypeWarning mixed types means in pandas read_csv, how it splits one column into ints and strings, and three ways to fix it, with counts.
7 min read
CSV
Using pandas read_csv for Files That Do Not Behave
Make pandas read_csv cope with title lines, comments, European numbers, odd NA markers, short or long rows and the wrong encoding, one option at a time.
9 min read
CSV
Writing a CSV File With Python and Keeping Quotes Correct
Write CSV in Python with csv.writer and DictWriter: newline='', the quoting modes, how None, floats, dates and bytes are written, and what Excel needs.
9 min read
CSV
Reading a CSV Into Dictionaries With csv DictReader
csv.DictReader turns each CSV row into a dict keyed by the header. See what it does with missing and extra fields, BOMs, duplicate headers and blank lines.
9 min read
CSV
Using the Python csv Module for Reading Files Line by Line
Read CSV in Python with csv.reader: open with newline='', handle quoted commas and multi-line fields, set the dialect, and stream millions of rows.
8 min read
Editor's pick
CSVGuide
Reading and Writing CSV Files in Python the Right Way
A working guide to CSV in Python: csv module, DictReader, pandas, polars and DuckDB measured on 2,000,000 rows, plus the encoding, quoting and type rules.
22 min read
CSV
Writing Query Results Back Out to CSV With DuckDB COPY
Export a DuckDB query to CSV with COPY. Control the delimiter, NULL text, quoting, date format and compression, and split output into one file per value.
9 min read
CSV
Joining Two CSV Files With SQL in DuckDB Without a Database
Join two CSV files on a key with plain SQL in DuckDB. Covers a 2M by 250k row join, keys that differ by case and spaces, left and anti joins, and type errors.
8 min read
Get new articles by email
We send one email when a new article is published. No spam.