Tag

how-to

590 posts

YAML

Preserving Comments and Order With ruamel yaml Round Trips

Edit a YAML file from Python and keep its comments, key order, quotes and block strings with ruamel.yaml round-trip mode, plus settings and caveats.

7 min read

XML

Reading Publisher Metadata in JATS XML for One Article

Pull the title with inline markup, authors with ORCID and affiliations, group authors, editors and the several dates out of one JATS article front matter.

7 min read

XML

Flattening a Product Catalog Feed With Repeating Fields

A shopping feed with a namespace, prices written as 12.50 EUR, optional sale prices and repeated shipping and image fields, flattened into three clean tables.

7 min read

XML

Joining XML Derived Tables to Your Own Data Safely

A join between parsed XML orders and a customer CSV silently dropped 1,280 lines and 3.2 million in revenue. Find the unmatched keys and normalize them.

6 min read

XML

Storing Raw XML in a Data Lake and Parsing It Later

Land source XML unchanged, compressed, partitioned by date and described by a manifest with a checksum, so parsing can be rerun when the rules change.

6 min read

XML

Using DuckDB to Read XML Through Python and Arrow Tables

Stream XML into Arrow batches with lxml, query them in DuckDB at once, or try the community webbed extension. 1.2 million lines in 7.9 s with a 53 ms query.

6 min read

XML

Compressing XML Archives and Reading Them In Place

A 127 MB XML file shrank to 14.8 MB with gzip, 15.4 MB with zstd and 10.8 MB with xz. Costs in seconds, and how to parse a gzip file without unpacking.

6 min read

XML

Parallel Processing of Split XML Files With Python Pool

Split a 127 MB XML file into four parts in 3.4 seconds, then process them in parallel for a 3.8 times speedup. Code, results and the limits.

6 min read

JSON

Indexing Hot JSON Properties for Dashboards in PostgreSQL

Expression, partial and GIN indexes on a 2.25 million row jsonb event table: a selective lookup fell from 65 ms to 5 ms, a 4 percent filter got slower.

6 min read

JSON

Extracting Properties Into Typed Columns Nightly From JSON

A nightly job that turns jsonb event properties into a typed facts table in PostgreSQL: safe casts, a watermark, idempotent inserts and a 47x speedup.

7 min read

JSON

Designing JSON Event Properties for Analysis From Day One

Four designs for the same 500,000 purchase events compared on size and query time: verbose nesting, short flat keys, hybrid columns and key/value arrays.

6 min read

JSON

Parsing a Package Registry Metadata File Where Keys Are Data

Read a package metadata document where version numbers are keys: json_each, DuckDB MAP inference, semver sorting, dist-tags versus max, and the time map.

7 min read