CSV
Archiving Processed CSV Files Automatically After Import
Move each CSV out of the inbox only after its load commits, compress and verify it into dated folders, and expire old files with a clear retention rule.
7 min read
Tag
automation
40 posts
CSV
Building Idempotent CSV Imports That Are Safe to Run Twice
Four ways to make a CSV import safe to rerun, tested on one three-row file run twice: upsert on a key, row hash, replace by batch and a content-hash ledger.
7 min read
CSV
Loading a CSV From S3 Into a Database Without a Local Copy
Four ways to load a CSV stored in S3 into a database: a streamed pipe, the RDS aws_s3 extension, DuckDB reading s3:// directly, and a download-then-load job.
7 min read
CSV
A Python Script That Loads a Daily CSV Into PostgreSQL
A complete psycopg 3 script that stages a daily CSV with COPY, merges it with an upsert, records a content hash, and is safe to run again after a failure.
8 min read
CSVGuide
Automating CSV Imports Into a Database Without Surprises
Build a CSV import job that runs unattended: idempotent loads, one transaction, reject files, an archive, a ledger, locking, alerts and a safe backfill.
21 min read
XML
Editing XML Files in Place From a Script Without Damage
Change a value, add an attribute and delete an element in XML from a script with xmlstarlet or lxml, and compare with sed on diffs, comments and escaping.
6 min read
CSV
Automating a Nightly CSV Export With cron and psql
A cron export that cannot publish a half-written file: lock, temporary name, row-count check, atomic rename. Real cron failed first, on the environment.
10 min read
YAML
Helm Values Files Without Copy Paste Using Layered Values
Layer Helm values so production overrides only what differs: maps merge, lists replace, --set wins. Rendered with helm template and checked with kubeconform.
7 min read
ExcelGuide
A Practical Guide to Automating Excel Work Without Clicking
How to move recurring Excel work into scripts: choosing VBA, Python, Office Scripts or Power Query, scheduling, merging, splitting, protecting and testing.
17 min read
YAML
Naming Keys Consistently in YAML Config Files Everywhere
Pick one key style, enforce it with a JSON Schema pattern and fix old files with a yq command that renames every key. Real output for each step.
6 min read
YAML
Validating Config YAML With JSON Schema in CI Pipelines
Write a JSON Schema for a YAML config, run check-jsonschema in CI, get every error with a path, and know that the loader decides what NO and 3.10 are.
7 min read
YAML
Environment Specific YAML Config Without Copy and Paste
Keep one base file and small overrides per environment, merge them in order and print the effective config with the source of each value. Tested layers.
6 min read