CSV
Choosing Between csvkit and Miller for Everyday CSV Work
csvkit and Miller timed on the same 200,000-row file. Type inference makes csvkit's sort, stat and JSON commands 100 times slower and can rewrite your dates.
6 min read
Topic
CSV
The format everyone exports and nobody specifies. Delimiters, quoting, encodings and loading it into a database intact.
CSV
Datasette for Publishing CSV as a Searchable Site in Minutes
Turn a CSV into a browsable, searchable, filterable website with sqlite-utils and Datasette, with a JSON and CSV API, facets and full-text search built in.
7 min read
CSV
Polars for Large CSV Processing Beyond Memory Limits
Use Polars lazy scanning to filter and group a large CSV without loading it whole, set column types so zip codes stay text, and write Parquet as you go.
6 min read
YAML
Round Tripping YAML Without Losing Comments or Order
Edit a YAML file in code and keep its comments, anchors and quotes. PyYAML drops them, ruamel.yaml and yq keep them, and each changes some whitespace.
6 min read
CSV
Using sqlite utils to Load CSV Fast Into a SQLite Database
Load a CSV into SQLite with one sqlite-utils command, set a primary key, control type detection, add full-text search, and see what 2 million rows cost.
6 min read
CSV
CSVLint and Other Validators for CSV Files Compared
Four CSV validators run on the same broken files: qsv validate, csvkit csvclean, CSVLint and Frictionless. What each reports, how it exits and which to use.
7 min read
CSV
VisiData for Exploring CSV Interactively in the Terminal
Open a CSV in VisiData to scroll, sort, filter and profile it from the keyboard, build frequency tables, and save results, plus its batch mode for scripts.
7 min read
CSV
Using Mlr Miller for Records and Pipelines on the Shell
Miller (mlr) treats CSV rows as named records and chains verbs with then. Real runs for sorting, grouping, new columns, JSON conversion and splitting files.
6 min read
CSV
Using xsv and qsv Side by Side on the Command Line
xsv and qsv share a command set, and qsv adds an index, sniffing, validation and more. See the overlap, the real timings with an index, and the type trap.
7 min read
CSVGuide
CSV Tools Worth Knowing and When to Reach for Each One
A field guide to CSV tools: csvkit, qsv, Miller, DuckDB, sqlite-utils, Polars, pandas and VisiData, run on the same 2 million row file with timings.
18 min read
CSV
Handling CSV With Comment Lines and Metadata Headers
Some CSV files carry lines starting with a hash that are notes, not data. Skip them safely, because a comment option can also cut values like Suite #4.
6 min read
CSV
Detecting and Handling Embedded Headers Repeated Mid File
A CSV that repeats its header after every page or section breaks type conversion. Find the repeated lines, drop them safely and check nothing real went too.
6 min read
Get new articles by email
We send one email when a new article is published. No spam.