XML
Extracting Time Series From XML Sensor Files With Pandas
Turn XML sensor readings into a pandas time series, resample to hourly means, find a 50 minute gap and a silent switch from Celsius to Fahrenheit.
6 min read
Tag
how-to
590 posts
CSV
Converting CSV to JSON With Python Without Losing Types
Convert CSV to JSON with the csv module, then decide on types: zip stays text, money stays exact. pandas gave 2134.0 for 02134; converters took 0.4 to 36 s.
9 min read
JSONGuide
Validating JSON With JSON Schema in Real Team Projects
A working guide to JSON Schema: write a schema, run it in Python and Node, handle formats, refs and versions, and check JSON stored in a database.
19 min read
CSVGuide
Converting CSV to JSON XML YAML and Excel in Practice
Convert one awkward CSV to JSON, JSON Lines, XML, YAML, Excel, Parquet, Markdown and SQL, read each back and see the cost on 193,708 rows.
22 min read
XML
Counting Records by Attribute in a Large XML Export
Count 400,000 orders by status in a 132 MB XML file with grep in 0.4 s, lxml iterparse in 2.0 s or xmlstarlet in 5.1 s, and know when each can be trusted.
6 min read
CSV
Automating a Nightly CSV Export With cron and psql
A cron export that cannot publish a half-written file: lock, temporary name, row-count check, atomic rename. Real cron failed first, on the environment.
10 min read
JSON
Reading JSON Lines With pandas Without the Usual Surprises
Read JSON Lines into a pandas DataFrame with read_json(lines=True): dtypes, chunking, nested fields, big integers, bad lines and the pyarrow engine.
6 min read
Excel
Publishing Database Results to a Sheet Automatically
Publish database results to a Google Sheet on a schedule: replace a range safely, mark it generated, stamp the update time and alert on failure.
8 min read
CSV
Exporting CSV From the Database Straight to a Gzip File
Pipe a PostgreSQL export into gzip or zstd: 130 MB became 42 MB in 4.6 s, with six settings compared and the pipeline that exits 0 after a failed export.
8 min read
XML
Summing Values Across Thousands of XML Invoices by Currency
Total 5,000 XML invoice files in under a second with Python, xmlstarlet or grep, group by currency, and catch malformed files and missing totals.
6 min read
CSV
Writing Dates and Booleans to CSV That Others Can Read
One timestamp exported as 09:30+00, 16:30+07 and 04:30-05, and dd.mm.yyyy that pandas read as 3 January. Fix dates and booleans in the query, not the session.
9 min read
JSON
Compressing JSON Lines for Storage and Reading Them Back
Compare gzip, zstd and xz on a 36 MB JSON Lines file for size and speed, and read the result back with jq, DuckDB, pandas and Python without unpacking.
6 min read