CSV
Background Jobs for Large CSV Exports That Do Not Time Out
Move big CSV exports into a job: chunk by key, write to storage, send an expiring link, track progress and clean up. Keyset chunks beat OFFSET.
8 min read
Tag
export
28 posts
CSV
Streaming a CSV Export From a Web App Without Memory Spikes
Stream a CSV download instead of building it in memory: one million rows took 336 MB as an array and 2 MB streamed. Headers, buffering and limits.
7 min read
PostgreSQL
Exporting JSON Lines From a Query in PostgreSQL Safely
Export JSON Lines from a PostgreSQL query with row_to_json or to_jsonb and COPY TO STDOUT, handle quoting, types, nulls, large results and compression.
6 min read
CSV
Scheduling CSV Reports From SQL With cron Reliably
A cron job that writes a daily CSV report from SQL: wrapper script, atomic output, a lock against overlap and the environment traps that bite at 2 a.m.
7 min read
CSV
Automating a Nightly CSV Export With cron and psql
A cron export that cannot publish a half-written file: lock, temporary name, row-count check, atomic rename. Real cron failed first, on the environment.
10 min read
CSV
Exporting CSV From the Database Straight to a Gzip File
Pipe a PostgreSQL export into gzip or zstd: 130 MB became 42 MB in 4.6 s, with six settings compared and the pipeline that exits 0 after a failed export.
8 min read
CSV
Writing Dates and Booleans to CSV That Others Can Read
One timestamp exported as 09:30+00, 16:30+07 and 04:30-05, and dd.mm.yyyy that pandas read as 3 January. Fix dates and booleans in the query, not the session.
9 min read
CSV
Exporting Large Tables to CSV Without Running Out of Memory
Export a 2,000,000 row table to CSV six ways: peak memory ranged from 11 MB to 2 GB, time from 1.6 s to 14 s. fetchall and pandas are the ones that run out.
9 min read
CSV
Exporting a CSV With Headers From SQLite on the Command Line
Export SQLite to CSV with a header row from the sqlite3 shell: CRLF endings by default, NULL against empty string, types and 2,000,000 rows in 3.5 s.
10 min read
CSV
Exporting a MongoDB Collection to CSV With mongoexport
mongoexport writes CSV only if you name the fields. See arrays, ObjectIds, nulls, the empty file and exit 0 on a typo, and 500,000 documents in 2.7 s.
10 min read
CSV
Exporting an Oracle Query to CSV With SQLcl From the Shell
Export an Oracle query to a quoted CSV with SQLcl: set the CSV format, spool to a file, fix date and number formats, and run it from a shell script.
8 min read
CSV
Exporting SQL Server Results to CSV With bcp and sqlcmd
How to export a SQL Server query to a real CSV with bcp: add the header row, quote fields yourself, fix dates and NULLs, and pick the right encoding.
10 min read