Topic

Scripting

Shell one-liners and small scripts that turn your repeated database chores into a single command.

CSV

Splitting a CSV by Column Value Into Separate Files

Split a CSV into one file per column value with awk, qsv, Miller or DuckDB, keep the header in every file, check the counts and avoid unsafe file names.

9 min read

CSV

Removing Duplicate Rows From a CSV With Shell Tools

Remove duplicate CSV rows with sort -u, awk, qsv and Miller. See what each does to the header and the order, which row it keeps, and why awk drops a good row.

9 min read

CSV

Sorting a CSV File by Column in the Terminal Correctly

Sort a CSV by column from the shell without losing the header or reading text as numbers. Compare sort, qsv, Miller, csvsort and DuckDB on 2,000,000 rows.

9 min read

CSV

Counting Rows and Unique Values in a CSV From the Shell

Count records and distinct values in a CSV from the shell with wc, sort, uniq, qsv and Miller, and see why wc -l and messy spellings give wrong counts.

7 min read

CSV

Using qsv for Fast CSV Statistics and Slicing of Big Files

qsv counts, profiles, searches, slices and sorts CSV files in fractions of a second. See count, stats, frequency, slice and index run on 2,000,000 rows.

9 min read

CSV

Filtering CSV Rows From the Command Line With Miller

Filter, sort, compute and summarize CSV from the shell with Miller's filter, put and then-chains, and see how it treats empty cells, leading zeros and sums.

8 min read

CSV

Getting Started With csvkit for Everyday CSV Tasks

csvkit gives you csvcut, csvgrep, csvlook, csvstat, csvsort and csvsql with real CSV parsing. See the everyday commands, inference traps and speed cost.

8 min read

CSV

Using awk on CSV Files and Where It Gets Dangerous

awk -F, sums and groups a CSV in one pass when nothing is quoted. See the cases where it silently returns wrong fields, and what FPAT does and does not fix.

8 min read

CSV

Selecting CSV Columns With cut and Why Quotes Break It

cut -d, -f picks CSV columns until a quoted comma shifts every field after it. See the failure, a guard that detects it, and CSV-aware replacements.

8 min read

CSV

Previewing a CSV in the Terminal With column and less

Look at a CSV in the terminal with head, column -t, less -S, csvlook, qsv table and Miller, and learn where each one misleads you on quotes and encodings.

8 min read

Working With CSV Files in the Shell Without Opening Excel

A practical guide to CSV in the shell: coreutils, awk, csvkit, Miller, qsv and DuckDB compared on 2,000,000 rows, with the quoting rule and a triage script.

22 min read

CSV

Python CSV Newlines and the Blank Line Bug on Windows

Why a CSV written by Python shows a blank line after every row on Windows, how newline='' prevents it, and how to check the bytes. Reproduced without Windows.

8 min read

Get new articles by email

We send one email when a new article is published. No spam.