JSON
Finding One Bad Line in a Huge JSON Lines File Fast
Locate the malformed record in a large JSON Lines file with jq, Python, awk and DuckDB, why parser line numbers drift, and how to split good from bad.
7 min read
Tag
error-fix
93 posts
CSV
Testing a CSV Export for Truncated Last Lines in Python
A cut-off CSV loaded in pandas without a warning, last row 200000,Ana,Smith,...,2. Check the final byte, the last record's width, quotes, row count and gzip.
11 min read
XML
XML Versus CSV for Exchanging Tabular Data Between Systems
A partner asks for XML for a flat table. Measured on 100,000 rows, XML was 2.7 times the CSV size, though gzip nearly erased the gap. When each is right.
6 min read
CSV
Detecting Columns That Changed Type Between CSV Deliveries
April's load failed on qty 1.5, but six columns had drifted. Compare per-column kinds of value between two CSV deliveries, not just inferred types.
11 min read
JSON
Reading JSON From an API With requests and Handling Errors
response.json() raised JSONDecodeError for an empty body, a 204, an HTML 502 and a BOM. Check status first, set a timeout and retry with a Session, as tested.
8 min read
CSV
Fixing Numbers With Thousands Separators in a CSV File
A comma inside a number breaks int(), shifts columns when unquoted, and turns European 1.200 into 1.2. Parse strictly, set thousands= and decimal=, and test.
8 min read
PostgreSQL
Why the Planner Estimates the Wrong Number of Rows
Why the PostgreSQL planner estimates the wrong number of rows: stale statistics, skew, correlated columns, expressions and parameters, with the fix for each.
7 min read
CSV
Fixing Mixed Date Formats in a CSV Column Without Guessing
Four formats in one date column, 959 values that are not dates and 15,642 that could be read two ways. Parse with explicit formats and report what you cannot.
9 min read
CSV
Fixing Garbled Characters in a Downloaded CSV From a Web App
A CSV from your web app opens as Zoë in Excel. Check the bytes, the charset header and the BOM the server sends, then fix Express, Blobs and file names.
8 min read
CSV
Why Splitting CSV by Comma in JavaScript Goes Wrong
line.split(',') breaks on quoted commas, quotes, line breaks and CRLF. See the failures, why the regex fix is slow, and a 30 line parser that works.
7 min read
XML
Debugging an XSLT Transform That Outputs Nothing or Too Much
Five stylesheets that output nothing or the whole file as text, with the cause of each: a namespace, a built-in rule, an absolute path, and a literal id.
7 min read
CSV
Python CSV Error Field Larger Than Field Limit and the Fix
Fix _csv.Error field larger than field limit (131072). Tell a genuinely large cell from an unclosed quote, raise the limit safely and find the bad record.
8 min read