CSV
VisiData for Exploring CSV Interactively in the Terminal
Open a CSV in VisiData to scroll, sort, filter and profile it from the keyboard, build frequency tables, and save results, plus its batch mode for scripts.
7 min read
Tag
cli
47 posts
CSV
Using Mlr Miller for Records and Pipelines on the Shell
Miller (mlr) treats CSV rows as named records and chains verbs with then. Real runs for sorting, grouping, new columns, JSON conversion and splitting files.
6 min read
CSV
Using xsv and qsv Side by Side on the Command Line
xsv and qsv share a command set, and qsv adds an index, sniffing, validation and more. See the overlap, the real timings with an index, and the type trap.
7 min read
CSVGuide
CSV Tools Worth Knowing and When to Reach for Each One
A field guide to CSV tools: csvkit, qsv, Miller, DuckDB, sqlite-utils, Polars, pandas and VisiData, run on the same 2 million row file with timings.
18 min read
XML
Splitting a Big XML File by Repeating Element Safely
Split a 36.6 MB XML file into eight valid files of 25,000 orders with iterparse in 1.5 seconds at 21 MB, and see why split -l produces broken XML.
6 min read
XML
Merging Several XML Files Into One Valid Document Safely
Why cat breaks XML, and three ways that work: XInclude with xmllint, XSLT with document(), and lxml, with the xml:base attribute side effect of XInclude.
7 min read
XML
Counting Nodes in XML From the Shell Without Miscounting
grep counts lines, not elements. Count XML elements correctly with xmllint, xmlstarlet and a streaming parser, and see why grep gave 20000 for 49819 lines.
6 min read
YAML
Converting YAML to JSON With yq and Back Again Cleanly
yq converts YAML to JSON and back in one command. The same file gave different JSON in yq and PyYAML, and a big integer made yq fail. Flags and traps.
6 min read
XML
Using yq to Read XML From the Command Line and Pipelines
Two different tools are called yq. Both read XML and print JSON or YAML, with different prefixes for attributes. See each one's output and which you have.
6 min read
XML
Editing XML Files in Place From a Script Without Damage
Change a value, add an attribute and delete an element in XML from a script with xmlstarlet or lxml, and compare with sed on diffs, comments and escaping.
6 min read
XML
Extracting Values With xmlstarlet From the Shell Fast
Select values, make delimited lines, count nodes, list element paths and sort with xmlstarlet sel, and avoid the escaped ampersand in its default text output.
6 min read
XML
Formatting XML With xmllint for Readable Diffs and Reviews
Pretty-print one-line XML with xmllint, set the indent, strip blank text for stable diffs, get canonical form and change encoding, and see what it alters.
7 min read