CSV
Scheduling CSV Reports From SQL With cron Reliably
A cron job that writes a daily CSV report from SQL: wrapper script, atomic output, a lock against overlap and the environment traps that bite at 2 a.m.
7 min read
Topic
Migrations
Change the schema while the site is live: expand-and-contract, backfills, and moving data between engines.
CSV
Logging Rejected Rows From a CSV Import for Review
Capture every row an import refuses, with its line, column and reason, in a form a person can fix and replay. A DuckDB example with real reject tables.
6 min read
CSV
Handling a CSV Provider Who Changes Columns Without Warning
A provider adds, renames or reorders columns and your load breaks or, worse, succeeds. A header check that names the change, aliases and a policy per case.
7 min read
CSV
Seeding a Database From CSV Files With dbt Seeds Safely
Load small reference CSVs with dbt seeds from version control, set column types so zip codes keep their zeros, and learn when seeds are the wrong tool.
6 min read
CSV
Loading CSV Files in Airflow With a Plain Python Task
Load a daily CSV in Airflow with one plain Python task instead of a chain of operators: idempotent by date, safely retried and testable outside Airflow.
7 min read
CSV
Watching a Folder for New CSV Files and Importing Them
Poll a folder for new CSV files and import each only once it has stopped changing. Why file events fire too early, a stable-file watcher and inotify on Linux.
7 min read
CSV
Archiving Processed CSV Files Automatically After Import
Move each CSV out of the inbox only after its load commits, compress and verify it into dated folders, and expire old files with a clear retention rule.
7 min read
CSV
Building Idempotent CSV Imports That Are Safe to Run Twice
Four ways to make a CSV import safe to rerun, tested on one three-row file run twice: upsert on a key, row hash, replace by batch and a content-hash ledger.
7 min read
CSV
Loading a CSV From S3 Into a Database Without a Local Copy
Four ways to load a CSV stored in S3 into a database: a streamed pipe, the RDS aws_s3 extension, DuckDB reading s3:// directly, and a download-then-load job.
7 min read
CSVGuide
Automating CSV Imports Into a Database Without Surprises
Build a CSV import job that runs unattended: idempotent loads, one transaction, reject files, an archive, a ledger, locking, alerts and a safe backfill.
21 min read
JSONGuide
Pulling JSON From APIs Into a Database Reliably in Python
A working guide to loading JSON APIs into a database: raw pages, pagination, rate limits, upserts, cursors, shape checks and retries, with a tested loader.
17 min read
YAML
Documenting Datasets in YAML and Catching Schema Drift
Describe a dataset in YAML, render a Markdown table from it and compare it with the live table to catch columns added or dropped without a doc.
6 min read
Get new articles by email
We send one email when a new article is published. No spam.