CSV
Testing a CSV Export for Truncated Last Lines in Python
A cut-off CSV loaded in pandas without a warning, last row 200000,Ana,Smith,...,2. Check the final byte, the last record's width, quotes, row count and gzip.
11 min read
Tag
quality
52 posts
CSV
Detecting Columns That Changed Type Between CSV Deliveries
April's load failed on qty 1.5, but six columns had drifted. Compare per-column kinds of value between two CSV deliveries, not just inferred types.
11 min read
CSV
Validating a CSV File Against a Lookup Table With SQL
Find CSV values missing from a lookup table with an anti-join, keep line numbers right, avoid the NOT IN trap with NULLs and see a foreign key's error.
10 min read
CSV
Checking That Every Date in a CSV File Parses Correctly
Check every date in a CSV with a strict parse and a round trip, then ranges. strptime accepted 2025-3-4, DuckDB cast kept the date of 2025-03-14 10:30.
10 min read
CSV
Finding Duplicate Keys in a CSV File Before an Import
Find repeated ids in a CSV before a primary key rejects the load, with SQL, pandas and uniq -d, line numbers and an exact versus conflicting split.
9 min read
CSV
Validating CSV Columns With Python and pydantic Models
Validate each CSV row against a pydantic model with types, literals, patterns and a cross-field rule: 193,708 rows in 0.95 s, with file line numbers.
10 min read
CSV
Writing a Schema for a CSV With Frictionless Data in Python
Infer a Table Schema, fix it by hand, add unique, enum and pattern constraints and validate with Frictionless: 10 errors for 8 faults in 193,708 rows.
11 min read
CSV
Checking Row and Column Counts of a CSV Before Loading
How many rows does this CSV have? wc -l, awk, csv, pandas and DuckDB gave 7, 8, 7, an error, 4 and 8 on one small file. Count records and field widths.
10 min read
CSVGuide
How to Validate a CSV File Before You Trust the Data
Validate a CSV delivery in five layers, bytes, structure, schema, values and keys, with line numbers and exit codes: 193,708 rows checked in about a second.
21 min read
XML
Testing Feed Compatibility Before a Schema Change Ships
Which XSD edits break old documents, which break old consumers, and a script that tests six changes both ways against saved samples before you ship.
6 min read
XML
Choosing Between XSD RELAX NG and Schematron for XML Checks
The same order checked three ways: XSD and RELAX NG passed a wrong total and a negative price, Schematron caught both, only RELAX NG took reordered children.
7 min read
XML
Common XSD Validation Errors Explained With the Fix for Each
Ten XSD validation messages from libxml2, exactly as printed, what each one means in the document and the smallest change that makes it validate.
7 min read