JSON
Compressing JSON Lines for Storage and Reading Them Back
Compare gzip, zstd and xz on a 36 MB JSON Lines file for size and speed, and read the result back with jq, DuckDB, pandas and Python without unpacking.
6 min read
Tag
cli
47 posts
JSON
Splitting a JSON Lines File Into Chunks Without Breakage
Split a JSON Lines file by line count, size or key with split and awk, keep every record whole, gzip chunks on the fly and process them in parallel.
7 min read
CSV
Exporting an Oracle Query to CSV With SQLcl From the Shell
Export an Oracle query to a quoted CSV with SQLcl: set the CSV format, spool to a file, fix date and number formats, and run it from a shell script.
8 min read
CSV
Exporting SQL Server Results to CSV With bcp and sqlcmd
How to export a SQL Server query to a real CSV with bcp: add the header row, quote fields yourself, fix dates and NULLs, and pick the right encoding.
10 min read
JSONGuide
Using jq to Slice Up JSON in the Terminal Reliably
A tested guide to jq 1.8 for slicing JSON in the terminal: filters, streams, CSV, grouping, logs, safe variables, in-place edits and number quirks.
17 min read
JSON
Pretty Printing JSON in Python and From the Terminal
Pretty print JSON with json.dumps(indent), python -m json.tool and pprint, keep non-ASCII readable, sort keys and collapse short arrays.
6 min read
CSV
Reading CSV Files in Bash Without Breaking on Quotes
A bash read loop on a CSV breaks on quoted commas, backslashes, a missing last newline and empty tab fields. Let a CSV parser emit safe records first.
10 min read
CSV
Splitting a CSV by Column Value Into Separate Files
Split a CSV into one file per column value with awk, qsv, Miller or DuckDB, keep the header in every file, check the counts and avoid unsafe file names.
9 min read
CSV
Removing Duplicate Rows From a CSV With Shell Tools
Remove duplicate CSV rows with sort -u, awk, qsv and Miller. See what each does to the header and the order, which row it keeps, and why awk drops a good row.
9 min read
CSV
Sorting a CSV File by Column in the Terminal Correctly
Sort a CSV by column from the shell without losing the header or reading text as numbers. Compare sort, qsv, Miller, csvsort and DuckDB on 2,000,000 rows.
9 min read
CSV
Counting Rows and Unique Values in a CSV From the Shell
Count records and distinct values in a CSV from the shell with wc, sort, uniq, qsv and Miller, and see why wc -l and messy spellings give wrong counts.
7 min read
CSV
Using qsv for Fast CSV Statistics and Slicing of Big Files
qsv counts, profiles, searches, slices and sorts CSV files in fractions of a second. See count, stats, frequency, slice and index run on 2,000,000 rows.
9 min read