JSON
Handling a Weather API With Nested Arrays of Measurements
Weather APIs return columns as parallel arrays. Turn them into rows with pandas and DuckDB, handle nulls and unequal lengths, and keep units and zones.
6 min read
Tag
python
134 posts
JSON
Cleaning a Twitter Style Archive File That Is Not Quite JSON
Clean a social media archive that is JavaScript, not JSON: strip the assignment, join parts, fix string numbers and dates, unescape HTML and write Parquet.
6 min read
JSON
Flattening a Shopify Style Order Export Into Tables
Flatten a Shopify style export of 20,000 orders into orders and line items with pandas and DuckDB: money as strings, null customers, tags and zones.
7 min read
JSON
Fixtures in JSON for Database Tests With PostgreSQL
Load JSON fixtures into PostgreSQL for tests with jsonb_populate_recordset: 25,000 rows in 0.21 s, bad fixtures caught by constraints, one typo missed.
6 min read
JSON
Contract Testing JSON APIs With an OpenAPI Schema Tool
Test a JSON API against its OpenAPI contract with Schemathesis: 1,903 generated requests found a 500 and undocumented statuses, and missed one bug.
6 min read
CSV
Making Quick Charts From a CSV With Python and DuckDB
Make quick charts from a large CSV: aggregate in DuckDB, plot with matplotlib or draw text bars in the terminal, and keep the charts honest.
6 min read
CSV
Checking if a CSV Column Is Normally Distributed With Python
Test whether a CSV column follows a normal distribution using skew, kurtosis, percentiles and Shapiro-Wilk, and why a huge file always fails the test.
7 min read
CSV
Bulk Loading a CSV File Into DynamoDB Tables Properly
Two ways to load a CSV into DynamoDB: the S3 import that makes a table of string attributes, and a boto3 batch writer with Decimal numbers.
7 min read
XML
Calling a SOAP Service From Python With zeep and a WSDL
Call a SOAP service from Python with zeep: load the WSDL, call an operation, get typed results, handle a Fault, see the request envelope,.
7 min read
JSON
Mock Data Generators for JSON Tests Demos and Load
Generate JSON test data with Faker, Hypothesis, SQL and jq: seeds for repeatable runs, collisions, and a property test that found real bugs.
6 min read
JSON
Comparing JSON Ignoring Volatile Fields in Python Tests
Compare JSON documents in tests while ignoring ids and timestamps: strip keys, DeepDiff exclusions, placeholders that keep relations, jq del, float tolerance.
6 min read
JSON
Snapshot Testing JSON Output Without Flaky Failures
Snapshot tests for JSON in pytest with syrupy and Node's test runner: what a snapshot stores, why random ids break it, and matchers and scrubbing that fix it.
6 min read