Config Driven Pipelines and Their Limits With Numbers

A config driven pipeline moves the shape of a job out of code and into a file: a list of steps, what each runs, what it needs first. A runner reads the file, orders the steps and executes them. The appeal is that adding a step is a three-line YAML edit that a reviewer can read. The question this post answers is how far that scales, measured on two axes. The first is mechanical: what does a big file cost to load and order? The second is the one that actually ends up deciding: when does the file stop being the right home? It belongs to the cluster on YAML in data engineering workflows. The basic runner is built in declaring pipelines in YAML for data jobs.

The test input

Each pipeline is a list of steps with an id, a command and, for every step after the first, one dependency on an earlier step:

pipeline: big
steps:
  - id: s0
    run: python step.py 0
  - id: s1
    run: python step.py 1
    needs: [s0]

The generator gives step i one dependency chosen at random (seeded, so the run repeats) from the steps before it, which makes a wide, irregular graph and not a single chain. That matters: a chain is the easy case for the ordering code and the unrealistic one. The sizes are 100, 1,000, 10,000 and 50,000 steps. Few real pipelines have 50,000 steps in one file, and the point of including them is to find the shape of the curve.

Loading

Parsing is the step that grows, because every step creates several Python objects:

At 50,000 steps and 3.1 MB the C loader took 2,130 ms in a single run. Doubling the size roughly doubles the time, which is what you expect from a parser, and the cost per step is about 0.015 to 0.04 ms. The pure Python loader is six to eight times slower at each size. That ratio is the practical lesson: yaml.safe_load uses the pure Python path unless you ask for the C one, and a pipeline of 10,000 steps goes from about a fifth of a second to well over a second just by that choice. Use yaml.load(text, Loader=yaml.CSafeLoader) where libyaml is available, and fall back to SafeLoader when it is not (YAML parsing speed for large files has the broader comparison across parsers).

Memory is the second number to watch. Ten thousand steps are 599 KB of text and about 38 MB of Python objects at peak, roughly 67 times the file. That figure counts Python allocations as tracemalloc sees them; it is a lower bound on process memory. A runner that parses once at startup does not care. A service that parses a fresh copy per request does.

Validation and ordering

After loading, the runner checks that every needs refers to a step that exists, and orders the steps:

 steps  yaml KB  load ms  validate ms  sort ms
   100        5        2          0.0        0
  1000       57       20          0.1        2
 10000      599      243          2.1       21
 50000     3121     2130         12.4      172

Checking dependencies is a dictionary lookup per edge: 12 ms for 50,000 steps. Ordering uses graphlib.TopologicalSorter and takes 172 ms at the largest size, 21 ms at 10,000. Together they are under a tenth of the load time. This is the useful negative result: the graph algorithms people worry about are not where the time goes. If a runner feels slow on startup, profile the parse before you touch the scheduler.

Where the file runs out

The numbers say size is not the limit. A pipeline of 1,000 steps loads in under 20 ms. What stops people is a different thing, and it shows up in the same order in most teams:

1. Repetition. Twelve tables get the same three steps: extract, validate, load. The YAML now holds 36 steps that differ in one word. Anchors and merge keys help for a while (YAML anchors for repeated records).

2. Parameters. The steps differ by table, date and environment. People add ${TABLE} placeholders and a substitution pass (environment variable substitution in YAML config).

3. Conditions. "Skip the load on weekends." "Run the backfill only in production." The file grows an if: key with a string that the runner must interpret.

4. Loops. "For each partition." Now the file has a for_each: key.

5. Templating the YAML. At this point a Jinja pass generates the YAML, and the artifact is a text template that is neither valid YAML nor easy to review.

Each step is reasonable alone. Together they rebuild a programming language inside strings, without a debugger, a type checker or a stack trace. The signal to watch is the first if: or for_each: key: it means the file has started to contain logic, and logic is cheaper to write and test in code.

A concrete comparison: three steps in YAML take four lines (- {id: a, run: extract} and two more). The Python equivalent, a function per step and a decorator for dependencies, takes more characters and gains a name for every concept, tests per function and an editor that can rename. At three steps YAML wins. At three hundred steps that vary by table, a loop that generates them wins, and the YAML becomes the output and not the source.

A middle path

Keep YAML for what is data and put the logic in code that the YAML refers to by name:

• The file lists which steps exist and what order they have. It names a handler (handler: load_table) and passes plain parameters (table: orders).

• Code implements the handlers. They are ordinary functions with tests.

• A validation pass checks that every named handler exists, every parameter has the right type and no dependency is missing, before anything runs (validating YAML with JSON Schema).

• Repetition across tables is handled by a short generator that writes the YAML, or by a tables: list the runner expands, with one well-defined expansion rule documented in one place.

That keeps a reviewer able to read the pipeline's shape in a diff while keeping conditions where a debugger can reach them. Airflow, Dagster, Prefect and dbt all drew the line in different places, which is evidence that the line is real and that it is a design choice rather than a rule. Which one fits depends on how often the shape changes (YAML) against how often the behavior changes (code).

Measuring your own file

Do not trust these ratios for your own repository. A ten-line script reproduces the table: read the file, time yaml.load with both loaders at the real size, time your validation and your ordering, and time the part you suspect, which is usually a substitution or template pass and not the parse. Record the numbers next to the pipeline and rerun them when the file doubles. If the load time matters, there are two cheap options before you change anything else. Cache the parsed result keyed by the file's hash, so an unchanged file is not parsed again. Or split the file by area and load only what the run needs (splitting a large YAML config into several files).

When a config driven pipeline is the wrong design

The steps are mostly conditions. If the file is a tree of if: keys, the author is writing a program in the wrong place. Use code.

You need to debug it interactively. A traceback through a generic runner says run_step, not which of your 800 YAML entries failed. Code gives names and line numbers.

Few people edit it. The benefit of a config file is that non-programmers can change it. If every edit is by an engineer who also writes the handlers, the indirection is pure cost.

The file is the output of a generator. Then the generator is the source and the YAML is a build artifact. Commit the generator, and consider not committing the artifact.

You depend on exact timing. The numbers above are medians on one machine. A job with a latency budget needs measurements on its own hardware.

The loader choice is described in the PyYAML documentation, and the ordering function in the graphlib reference. Format a pipeline file with the YAML formatter.