CSV
Making a CSV Contract With Your Data Provider in Writing
What to agree with whoever sends you CSV files, as a document plus a manifest your code can check: six deliveries, six different rejections, 193,708 rows.
12 min read
Topic
Data Wrangling
Take messy CSV, JSON and spreadsheets and land them in tables without losing a row or a character on the way.
CSV
Testing a CSV Export for Truncated Last Lines in Python
A cut-off CSV loaded in pandas without a warning, last row 200000,Ana,Smith,...,2. Check the final byte, the last record's width, quotes, row count and gzip.
11 min read
CSV
Detecting Columns That Changed Type Between CSV Deliveries
April's load failed on qty 1.5, but six columns had drifted. Compare per-column kinds of value between two CSV deliveries, not just inferred types.
11 min read
CSV
Validating a CSV File Against a Lookup Table With SQL
Find CSV values missing from a lookup table with an anti-join, keep line numbers right, avoid the NOT IN trap with NULLs and see a foreign key's error.
10 min read
CSV
Checking That Every Date in a CSV File Parses Correctly
Check every date in a CSV with a strict parse and a round trip, then ranges. strptime accepted 2025-3-4, DuckDB cast kept the date of 2025-03-14 10:30.
10 min read
CSV
Finding Duplicate Keys in a CSV File Before an Import
Find repeated ids in a CSV before a primary key rejects the load, with SQL, pandas and uniq -d, line numbers and an exact versus conflicting split.
9 min read
CSV
Validating CSV Columns With Python and pydantic Models
Validate each CSV row against a pydantic model with types, literals, patterns and a cross-field rule: 193,708 rows in 0.95 s, with file line numbers.
10 min read
CSV
Writing a Schema for a CSV With Frictionless Data in Python
Infer a Table Schema, fix it by hand, add unique, enum and pattern constraints and validate with Frictionless: 10 errors for 8 faults in 193,708 rows.
11 min read
CSV
Checking Row and Column Counts of a CSV Before Loading
How many rows does this CSV have? wc -l, awk, csv, pandas and DuckDB gave 7, 8, 7, an error, 4 and 8 on one small file. Count records and field widths.
10 min read
CSVGuide
How to Validate a CSV File Before You Trust the Data
Validate a CSV delivery in five layers, bytes, structure, schema, values and keys, with line numbers and exit codes: 193,708 rows checked in about a second.
21 min read
CSV
Finding Rows That Do Not Match the Expected Pattern in a CSV
Check every column of a CSV against a regex and report failing rows with file line numbers. fullmatch beats match, and 2025-13-45 passes a date pattern.
9 min read
CSV
Fixing Numbers With Thousands Separators in a CSV File
A comma inside a number breaks int(), shifts columns when unquoted, and turns European 1.200 into 1.2. Parse strictly, set thousands= and decimal=, and test.
8 min read
Get new articles by email
We send one email when a new article is published. No spam.