CSV
Sorting a CSV File by Column in the Terminal Correctly
Sort a CSV by column from the shell without losing the header or reading text as numbers. Compare sort, qsv, Miller, csvsort and DuckDB on 2,000,000 rows.
9 min read
Library
Everything you need to level up your coding skills.
XML
Reading Government Open Data in XML With a USGS Feed
Read a real public QuakeML feed from the USGS: two namespaces, nested value elements, depth in meters, magnitude types, and checks that catch a wrong unit.
6 min read
YAML
Storing YAML Text in a Database Column and Its Pitfalls
What goes wrong when YAML is kept as text in a column: LIKE misses, equal documents compare unequal and garbage is accepted. The jsonb alternative fixes each.
6 min read
JSON
Indexing JSON Fields in SQLite With Expression Indexes
Index a field inside a SQLite JSON document with an expression index or a generated column, and see which query spellings use it and what it costs.
7 min read
Excel
Using the Excel Data Model for Larger Datasets in Practice
Load millions of rows into the Excel Data Model with Power Query, relate tables, build pivots on it, keep it small and know the limits of memory and bitness.
7 min read
CSV
Counting Rows and Unique Values in a CSV From the Shell
Count records and distinct values in a CSV from the shell with wc, sort, uniq, qsv and Miller, and see why wc -l and messy spellings give wrong counts.
7 min read
XML
Building Your Own XML Sitemap From a Table in Python and SQL
Generate sitemap files from a PostgreSQL table with a streaming writer: 117,002 URLs in 0.6 seconds, split at 50,000 per file, with an index and validation.
6 min read
CSV
Using qsv for Fast CSV Statistics and Slicing of Big Files
qsv counts, profiles, searches, slices and sorts CSV files in fractions of a second. See count, stats, frequency, slice and index run on 2,000,000 rows.
9 min read
Excel
Benchmarking Excel Readers in Python With a Fair Test
Compare openpyxl, calamine, pandas engines and DuckDB for reading xlsx: build test files, time runs, record peak memory and avoid common benchmark mistakes.
7 min read
JSON
Building JSON Results With json_group_array in SQLite
Build nested JSON from SQLite tables with json_group_array, json_group_object and json_object, and avoid the lost-subtype bug that turns arrays into strings.
7 min read
CSV
Filtering CSV Rows From the Command Line With Miller
Filter, sort, compute and summarize CSV from the shell with Miller's filter, put and then-chains, and see how it treats empty cells, leading zeros and sums.
8 min read
YAML
Exporting a Table to YAML From PostgreSQL With Python
Export a PostgreSQL table to YAML with psycopg and PyYAML. Fix the UUID error, keep numerics exact as strings and compare with a to_jsonb pipeline.
7 min read
Get new articles by email
We send one email when a new article is published. No spam.