JSON
Converting JSON to CSV With Python and pandas Safely
Convert JSON to CSV with pandas and the csv module, and avoid the traps: Python repr in cells, True and False, ragged keys, Excel encoding and formula cells.
8 min read
Tag
python
134 posts
YAML
Testing YAML Driven Jobs With pytest and Hypothesis
Test a YAML pipeline runner with pytest: order, unknown steps, cycles, duplicates, a Hypothesis property over random graphs and a check that the tests bite.
7 min read
YAML
Generating SQL From YAML Templates Safely With Bound Values
See an f-string builder turn a YAML job file into a dropped table, then fix it with bound values, quoted identifiers and an allow-list of tables.
6 min read
JSON
Generating a JSON Schema From Sample Data and Fixing It
Draft a JSON Schema from sample documents with genson, see what it infers and what it misses, and turn the draft into a contract with DuckDB as a cross-check.
7 min read
CSV
Converting CSV to Markdown Tables for Documentation
pandas, Miller and csvlook turn a CSV into a Markdown table, and each breaks on a pipe or a newline. pandas wrote 12.50 as 12.5 and csvlook 02134 as 2,134.
9 min read
CSV
Converting CSV to SQL INSERT Statements the Safe Way
Generate INSERT statements from a CSV without breaking on O'Neil or running a DROP TABLE hidden in a value. Batches of 1,000 loaded 193,708 rows in 1.6 s.
13 min read
CSV
Converting JSON to CSV When the Data Is Nested With Arrays
Flatten nested JSON to CSV with pandas, Python or DuckDB: dotted columns, a row per array item, the order that vanishes and arrays written as Python text.
10 min read
CSV
Converting Excel to CSV Without Losing Leading Zeros
pandas turned a text cell 02134 into 2134 and in2csv did too; qsv kept it. A number formatted 00000 loses its zeros everywhere unless you rebuild them.
10 min read
CSV
Converting CSV to an Excel Workbook With Formatting
Write a CSV to .xlsx with xlsxwriter: text zips, number formats, real dates and a frozen header. write() made =1+1 a formula; rows past 1,048,575 vanished.
8 min read
CSV
Converting CSV to YAML for Config Seeds Without Surprises
Hand-written YAML turned NO into false, 02134 into 1116 and 12:30 into 750; yaml.safe_dump quotes them. Dumping 193,708 records took 53 s, 15 s with libyaml.
9 min read
CSV
Converting CSV to XML With a Short and Safe Python Script
Convert CSV to XML with the standard library: invalid tag names and control characters need your own checks; a streaming writer does 193,708 rows in 22 MB.
10 min read
CSV
Converting CSV to JSON Lines for Streaming Pipelines
Convert CSV to JSON Lines, one object per line: a 13 MB streaming script, DuckDB in 0.3 s, files that split cleanly and a cut-off file that shows itself.
9 min read