YAML
Testing YAML Driven Jobs With pytest and Hypothesis
Test a YAML pipeline runner with pytest: order, unknown steps, cycles, duplicates, a Hypothesis property over random graphs and a check that the tests bite.
7 min read
Topic
Migrations
Change the schema while the site is live: expand-and-contract, backfills, and moving data between engines.
YAML
Config Driven Pipelines and Their Limits With Numbers
Measure what a YAML pipeline file costs to load, validate and order from 100 to 50,000 steps, then see where the file stops being the right place for logic.
7 min read
YAML
Generating SQL From YAML Templates Safely With Bound Values
See an f-string builder turn a YAML job file into a dropped table, then fix it with bound values, quoted identifiers and an allow-list of tables.
6 min read
YAML
Generating Table Schemas From YAML Definitions Safely
Generate CREATE TABLE statements from a YAML definition, quote identifiers so order and group work, and run the same DDL on PostgreSQL, SQLite and DuckDB.
6 min read
YAML
Defining Data Contracts in YAML That Fail the Build
Write a data contract as a YAML file, check a table against it with DuckDB, and fail the job on nulls, duplicates, bad values and drift, with tested output.
7 min read
YAMLGuide
YAML in Data Engineering Workflows and Where It Fits
Where YAML belongs in a data platform: tool config, contracts, schemas, pipelines, tests and docs, with a tested end to end example and the limits of each.
17 min read
Get new articles by email
We send one email when a new article is published. No spam.