CSV
Opening a CSV File That Is Too Large for Excel or Sheets
A worksheet holds 1,048,576 rows. When a CSV has more, filter, aggregate or split it first with DuckDB, split or csvkit, and hand Excel a file that fits.
6 min read
Topic
Performance
Why the query is slow, and what to change about it. Execution plans, statistics, caching and rewrites, with the numbers to prove it.
PostgreSQL
Tuning PostgreSQL for SSDs With random_page_cost and More
Tune PostgreSQL for SSDs: lower random_page_cost, set effective_io_concurrency, check changed plans, use per tablespace values and keep scans that win.
7 min read
JSON
Compressing JSON Lines for Storage and Reading Them Back
Compare gzip, zstd and xz on a 36 MB JSON Lines file for size and speed, and read the result back with jq, DuckDB, pandas and Python without unpacking.
6 min read
PostgreSQL
Comparing PostgreSQL Plans Before and After a Change
Compare PostgreSQL query plans before and after a change: capture both, control cache and noise, line up nodes and decide if the gain is real or warm cache.
7 min read
JSON
Splitting a JSON Lines File Into Chunks Without Breakage
Split a JSON Lines file by line count, size or key with split and awk, keep every record whole, gzip chunks on the fly and process them in parallel.
7 min read
PostgreSQL
Reading BUFFERS Output in EXPLAIN for PostgreSQL Queries
Read the BUFFERS output in PostgreSQL EXPLAIN: shared hit, read, dirtied and written, temp blocks, I/O timing, and what the numbers say about cache and disk.
7 min read
JSON
Faster JSON in Python With orjson and What Changes
orjson wrote a 35 MB document 6 times faster than json and parsed it 1.7 times faster. What it returns, what it rejects and what to check before migrating.
6 min read
XML
Benchmarking ElementTree lxml and iterparse on 127 MB
Four ways to sum 1.2 million order lines from a 127 MB XML file, with time and peak memory, plus the clearing pattern that keeps streaming flat.
6 min read
XML
How Much Memory a DOM Parser Needs for a Large XML File
Measured peak memory of ElementTree and lxml on 32 MB and 127 MB order files shows 11 to 15 times the file size, and how streaming cut it to 22 MB.
6 min read
XMLGuide
Parsing Big XML Files Fast Without Running Out of Memory
A measured guide to large XML files: memory multipliers, streaming, library choice, COPY into PostgreSQL, parallel splits, compression and indexes.
17 min read
PostgreSQL
Why the Planner Estimates the Wrong Number of Rows
Why the PostgreSQL planner estimates the wrong number of rows: stale statistics, skew, correlated columns, expressions and parameters, with the fix for each.
7 min read
PostgreSQL
Nested Loop Hash Join and Merge Join Compared in Plans
Nested loop, hash join and merge join compared in PostgreSQL: how each works, cost shape, memory and batches, plan clues and fixes when the choice is wrong.
7 min read
Get new articles by email
We send one email when a new article is published. No spam.