XML
Flattening a Product Catalog Feed With Repeating Fields
A shopping feed with a namespace, prices written as 12.50 EUR, optional sale prices and repeated shipping and image fields, flattened into three clean tables.
7 min read
Tag
etl
32 posts
XML
Storing Raw XML in a Data Lake and Parsing It Later
Land source XML unchanged, compressed, partitioned by date and described by a manifest with a checksum, so parsing can be rerun when the rules change.
6 min read
JSON
Extracting Properties Into Typed Columns Nightly From JSON
A nightly job that turns jsonb event properties into a typed facts table in PostgreSQL: safe casts, a watermark, idempotent inserts and a 47x speedup.
7 min read
PostgreSQL
Storing Uploaded Excel Files and Their Parsed Rows
Store uploaded Excel files and their parsed rows in PostgreSQL: keep the original, stage, track errors, dedupe by hash and link rows to uploads.
7 min read
JSON
Flattening a Shopify Style Order Export Into Tables
Flatten a Shopify style export of 20,000 orders into orders and line items with pandas and DuckDB: money as strings, null customers, tags and zones.
7 min read
JSON
Normalizing a Stripe Style Webhook Payload in DuckDB
Turn a Stripe style webhook log into tables with DuckDB: dedupe retries, zero-decimal currencies, invoice lines and the metadata map that hides gaps.
7 min read
XML
Incremental XML Feed Imports by Last Modified and ETag
Fetch a feed only when it changed: ETag and Last-Modified headers, a 304 response, a state table, and what to do when the server ignores them.
6 min read
JSONGuide
Pulling JSON From APIs Into a Database Reliably in Python
A working guide to loading JSON APIs into a database: raw pages, pagination, rate limits, upserts, cursors, shape checks and retries, with a tested loader.
17 min read
JSON
Normalizing a JSON Export Into Relational Tables Safely
Turn a blog export with authors, tags and comments into five PostgreSQL tables, with a tag junction table, merged author copies and checks against the source.
8 min read
JSON
Turning a Deeply Nested API Response Into Three Tables
Split a nested order response into customers, orders and order_items in PostgreSQL with keys and constraints, and see what the nested address belongs to.
7 min read
YAML
Config Driven Pipelines and Their Limits With Numbers
Measure what a YAML pipeline file costs to load, validate and order from 100 to 50,000 steps, then see where the file stops being the right place for logic.
7 min read
YAMLGuide
YAML in Data Engineering Workflows and Where It Fits
Where YAML belongs in a data platform: tool config, contracts, schemas, pipelines, tests and docs, with a tested end to end example and the limits of each.
17 min read