CSV
Logging Rejected Rows From a CSV Import for Review
Capture every row an import refuses, with its line, column and reason, in a form a person can fix and replay. A DuckDB example with real reject tables.
6 min read
Library
Everything you need to level up your coding skills.
CSV
Handling a CSV Provider Who Changes Columns Without Warning
A provider adds, renames or reorders columns and your load breaks or, worse, succeeds. A header check that names the change, aliases and a policy per case.
7 min read
CSV
Seeding a Database From CSV Files With dbt Seeds Safely
Load small reference CSVs with dbt seeds from version control, set column types so zip codes keep their zeros, and learn when seeds are the wrong tool.
6 min read
CSV
Loading CSV Files in Airflow With a Plain Python Task
Load a daily CSV in Airflow with one plain Python task instead of a chain of operators: idempotent by date, safely retried and testable outside Airflow.
7 min read
CSV
Watching a Folder for New CSV Files and Importing Them
Poll a folder for new CSV files and import each only once it has stopped changing. Why file events fire too early, a stable-file watcher and inotify on Linux.
7 min read
CSV
Archiving Processed CSV Files Automatically After Import
Move each CSV out of the inbox only after its load commits, compress and verify it into dated folders, and expire old files with a clear retention rule.
7 min read
CSV
Building Idempotent CSV Imports That Are Safe to Run Twice
Four ways to make a CSV import safe to rerun, tested on one three-row file run twice: upsert on a key, row hash, replace by batch and a content-hash ledger.
7 min read
CSV
Loading a CSV From S3 Into a Database Without a Local Copy
Four ways to load a CSV stored in S3 into a database: a streamed pipe, the RDS aws_s3 extension, DuckDB reading s3:// directly, and a download-then-load job.
7 min read
CSV
A Python Script That Loads a Daily CSV Into PostgreSQL
A complete psycopg 3 script that stages a daily CSV with COPY, merges it with an upsert, records a content hash, and is safe to run again after a failure.
8 min read
CSVGuide
Automating CSV Imports Into a Database Without Surprises
Build a CSV import job that runs unattended: idempotent loads, one transaction, reject files, an archive, a ledger, locking, alerts and a safe backfill.
21 min read
CSV
Limiting File Size and Row Count for CSV Uploads Safely
Pick CSV upload limits from measured parse and load speed, enforce each where it is cheapest, and reject bad shapes with a script that names the line.
7 min read
CSV
Why a Hostile CSV Upload Can Hurt Your Application Server
A CSV upload needs no exploit to take a server down: compressed bombs, one huge field, an unclosed quote and wide files, each with the limit that stops it.
7 min read
Get new articles by email
We send one email when a new article is published. No spam.