YAML
Diffing YAML Files Without Whitespace Noise or Reordering
diff -w hides YAML indentation changes that alter meaning. Normalize both files with yq sort_keys and compare as data to ignore layout and order only.
6 min read
Tag
cli
47 posts
YAML
Merging YAML Files From the Command Line With yq Safely
Deep-merge a base and an override with yq eval-all, choose whether lists replace or append, and check what a null or an empty value in the override does.
6 min read
JSON
Converting JSON to YAML on the Command Line Safely
Convert JSON to YAML with yq, Python's yq and a PyYAML one-liner, and see which strings get corrupted by YAML 1.1 readers: NO, yes, 12:30, 0123 and 1e3.
7 min read
YAML
Fixing YAML Indentation Errors With a Linter and Editor
A messy file with nine findings, reported by yamllint line by line, then repaired with yq and a project config. The workflow that finds and fixes indentation.
7 min read
CSV
Loading CSV Into MongoDB With mongoimport and Types
Load a CSV into MongoDB with mongoimport: what its type guessing does to zip codes, how to declare types, skip blanks and upsert on a key, with real runs.
7 min read
CSV
Loading CSV Into ClickHouse From the Command Line Fast
Load CSV into ClickHouse with clickhouse-client and FORMAT CSVWithNames, tune delimiter, errors and dates, and check a file first with clickhouse-local.
7 min read
CSV
Loading CSV Into BigQuery With bq load and Its Flags
Load a CSV from Cloud Storage into BigQuery with bq load: explicit schema, bad-record limits, quoted newlines, replace versus append and a job ID for reruns.
8 min read
CSV
Choosing Between csvkit and Miller for Everyday CSV Work
csvkit and Miller timed on the same 200,000-row file. Type inference makes csvkit's sort, stat and JSON commands 100 times slower and can rewrite your dates.
6 min read
CSV
Datasette for Publishing CSV as a Searchable Site in Minutes
Turn a CSV into a browsable, searchable, filterable website with sqlite-utils and Datasette, with a JSON and CSV API, facets and full-text search built in.
7 min read
JSON
Using JSON in Bash Scripts Safely With jq Arguments
Build and read JSON in Bash without breaking: jq --arg instead of string concatenation, newlines in values, exit codes, null as text and filter injection.
7 min read
CSV
Using sqlite utils to Load CSV Fast Into a SQLite Database
Load a CSV into SQLite with one sqlite-utils command, set a primary key, control type detection, add full-text search, and see what 2 million rows cost.
6 min read
CSV
CSVLint and Other Validators for CSV Files Compared
Four CSV validators run on the same broken files: qsv validate, csvkit csvclean, CSVLint and Frictionless. What each reports, how it exits and which to use.
7 min read