JSON
Flattening a Shopify Style Order Export Into Tables
Flatten a Shopify style export of 20,000 orders into orders and line items with pandas and DuckDB: money as strings, null customers, tags and zones.
7 min read
Topic
Data Wrangling
Take messy CSV, JSON and spreadsheets and land them in tables without losing a row or a character on the way.
JSON
Analyzing a GitHub Events Dump With DuckDB and SQL
Query a GitHub style events dump of 300,000 JSON lines in DuckDB: one struct for ten event types, truncated commit lists and three-state booleans.
7 min read
JSON
Normalizing a Stripe Style Webhook Payload in DuckDB
Turn a Stripe style webhook log into tables with DuckDB: dedupe retries, zero-decimal currencies, invoice lines and the metadata map that hides gaps.
7 min read
CSVGuide
Worked CSV Problems and the Patterns Behind Them Explained
Eight messy CSV files solved end to end with a check each: mixed dates, unquoted commas, multi-line notes, display formats, headers, bad quotes, footnotes.
18 min read
CSV
Loading CSV Into MongoDB With mongoimport and Types
Load a CSV into MongoDB with mongoimport: what its type guessing does to zip codes, how to declare types, skip blanks and upsert on a key, with real runs.
7 min read
CSV
Bulk Loading a CSV File Into DynamoDB Tables Properly
Two ways to load a CSV into DynamoDB: the S3 import that makes a table of string attributes, and a boto3 batch writer with Decimal numbers.
7 min read
CSV
Loading CSV Into Oracle With SQL Loader Control Files
Load a CSV into Oracle with SQL*Loader: a control file for quotes and dates, the bad and log files, exit codes, direct path and external tables.
7 min read
CSV
Loading CSV Into SQL Server With BULK INSERT Correctly
Load a CSV into SQL Server with BULK INSERT: FORMAT CSV for quoted fields, FIRSTROW, UTF-8, error files, server paths and permissions.
7 min read
CSV
Loading CSV Into ClickHouse From the Command Line Fast
Load CSV into ClickHouse with clickhouse-client and FORMAT CSVWithNames, tune delimiter, errors and dates, and check a file first with clickhouse-local.
7 min read
CSV
Loading CSV Into Snowflake and Handling Errors in COPY
Load CSV into Snowflake with a file format, a stage and COPY INTO, then read the errors: ON_ERROR, VALIDATION_MODE, VALIDATE and the 64 day skip rule.
7 min read
XML
Incremental XML Feed Imports by Last Modified and ETag
Fetch a feed only when it changed: ETag and Last-Modified headers, a 304 response, a state table, and what to do when the server ignores them.
6 min read
XML
Handling Compressed XML Downloads Without Surprises
A gzip file, gzip transfer encoding and unlabeled gzip behave differently in requests and urllib. Sniff the bytes, stream 36.6 MB at 21 MB, and guard bombs.
6 min read
Get new articles by email
We send one email when a new article is published. No spam.