CSV
The UTF8 BOM in CSV Files and When It Helps or Hurts
Three invisible bytes at the start of a CSV can fix Excel or break a header match. See which readers strip a BOM and how to add or remove one.
7 min read
Topic
Data Wrangling
Take messy CSV, JSON and spreadsheets and land them in tables without losing a row or a character on the way.
CSV
Why a CSV Shows Strange Characters Instead of Accents
Seeing odd letter pairs where accents belong means UTF-8 was read as a legacy encoding. See the byte level cause and a table of symptoms.
7 min read
CSV
How to Detect the Encoding of a CSV File Before You Load It
Four encoding detectors gave four different answers on the same six files. See which ones to trust, how to cross check them, and when only the sender knows.
7 min read
CSVGuide
Fixing Garbled Text in CSV Files and Keeping It Fixed
Why accents, euro signs and emoji break in CSV files, how to diagnose each symptom from the bytes, and one routine that converts, loads and checks a file.
21 min read
CSV
A Cheat Sheet of CSV Edge Cases to Test Every Parser With
Thirty six tiny CSV files that separate good readers from bad ones, with the right answer for each and the readers that missed it in my tests.
8 min read
CSV
TSV Versus CSV and When Tabs Are the Saner Choice for Data
TSV avoids most quoting because tabs rarely appear in text, but there are three TSV dialects and they disagree. See where each breaks and which to pick.
7 min read
CSV
How to Escape Quotes and Newlines Inside CSV Fields
Write CSV that survives every reader. Wrap fields with commas, quotes or line breaks, double inner quotes, and round trip test it against four readers.
9 min read
CSV
Why a CSV File Has No Types and What That Costs You
A CSV stores text, so every reader infers types. Measure what inference costs in time and memory on 2 million rows, and what it breaks: ZIP codes and dates.
8 min read
CSV
How to Tell What Delimiter an Unknown CSV File Uses
Find a file's delimiter in four steps, from looking at the bytes to a quote aware check, and see where Python, DuckDB and qsv sniffers disagree.
9 min read
CSV
Does a CSV File Need a Header Row and What If It Has None
A CSV has no flag that says whether row one is a header, so readers guess. Test four files, set the header explicitly, and handle duplicate names.
8 min read
CSV
CSV Line Endings and the CR LF Trap in Shells and Loaders
A CSV from Windows leaves an invisible carriage return on every last column. See it in a shell loop and MySQL, and fix it without damaging quoted fields.
8 min read
CSV
How CSV Quoting Works and Why Embedded Commas Break Things
See how a stray comma turns one column into two, what the four quoting states are, and why a backslash does not escape a quote in CSV the way it does in code.
8 min read
Get new articles by email
We send one email when a new article is published. No spam.