Structuring YAML Configuration Files That Last for Years

The configuration file you write today will outlive the person who wrote it, the first three versions of the application and probably the service it configures. What decides whether it is still understandable in five years is structure: where each kind of setting lives, how environments differ, where secrets are, how mistakes are caught and how the format itself changes. This guide describes a structure that holds up, shows it working in a small tested project, and explains the reasoning for each part so that you can adapt it. The detailed posts on each piece are linked as we go.

Why configuration rots

Three forces degrade configuration files, and every practice below answers one of them.

Copies diverge. The same settings appear in several files, one per environment or per service. A fix applied to one is forgotten in the others. Over a year the differences accumulate, none of them chosen, and the files are valid and wrong in different ways.

Mistakes are silent. A misspelled key is ignored. A number that should be a string is accepted. A duplicate key is resolved by the parser without telling you. A value like NO or 3.10 quietly becomes something else. The application starts, and the wrong behavior is found by a user.

Knowledge leaks away. Why is this timeout 30? Who owns this section? Is that flag still used? The answers lived in someone's head, and the file preserves only the values.

A structure that lasts keeps one copy of each fact, makes mistakes loud, and writes the reasons down in the file.

A layout

For an application with a few environments, a layout like this is enough:

config/
  base.yaml            # shared defaults, the source of truth
  env/
    dev.yaml           # only what differs in dev
    staging.yaml
    prod.yaml
schemas/
  app.json             # JSON Schema for the merged configuration
tools/
  yamlkit.py           # strict loader, env expansion, layered merge
  check_config.py      # merges each environment and validates it
.yamllint              # style rules

Four ideas are in that tree. The base holds everything shared, with a header comment that states the rules of the system (lists replace, null sets null). Each environment file holds only differences. The schema describes the merged configuration, the thing that actually runs. And a check merges every environment and validates it, so a bad edit fails in CI and not in production.

The base from the test project:

# Defaults shared by every environment. Overrides live in env/<name>.yaml.
# Lists replace (they are not appended); null removes nothing, it sets null.
schema_version: 1
env: base
server:
  host: localhost
  port: 8080
  tls: false
database:
  url: postgres://[email protected]/app
  password: ${DB_PASSWORD}      # resolved from the environment at start-up
  pool_size: 10
features: [search]
logging: {level: info}

and the production override, five lines:

env: prod
server: {host: app.example, tls: true}
database: {pool_size: 40}
logging: {level: warn}
features: [search, billing]

Production changes the host, turns on TLS, raises the pool size, quiets logging and extends the feature list. It says nothing about the port, the URL or the password, which therefore come from the base. When the base port changes, one edit changes it everywhere (environment specific YAML config without copy and paste goes through the layering and the "which file set this value" printout).

Secrets are references

Nothing secret appears in any layer. The base refers to ${DB_PASSWORD}, and the loader expands it from the environment after parsing the YAML, so a password with a colon, a space and a hash arrives intact and cannot change the structure of the file. A literal password removed from the file in a later commit stayed in Git history in a test: git log -S found both the commit that added it and the one that removed it, and a history scan found the value at the line where it was added. A committed secret is compromised, and the fix is rotation, with history cleanup second (keeping secrets out of YAML config files). Put a scanner in the pre-commit hook and in CI, and make the loader refuse a literal where a reference is expected.

Load strictly

The default loader forgives too much. A strict loader built on PyYAML's safe loader rejects duplicate keys, requires the top level to be a mapping, treats an empty file as an empty mapping and wraps parse errors with the file name and position. Module yamlkit.py in the project does that in about sixty lines, and it was tested with a duplicate key, a syntax error, a top-level list, an empty file and a missing environment variable, each of which produced a ConfigError naming the file (reading and writing YAML in Python safely has the module). Use safe_load or a subclass and never yaml.load with an unsafe loader, and if untrusted input can reach the loader, add limits on size and depth.

Validate the result

A schema turns the unspoken rules into checks. The test project's schema forbids unknown keys, requires the keys the application needs, constrains env to three values and the port to a range, requires pool_size to be an integer, and requires schema_version to be 1. The check script merges each environment file over the base, with secrets from a test environment, and validates the result:

for path in sorted(glob.glob("config/env/*.yaml")):
    name = os.path.basename(path)[:-5]
    cfg = yamlkit.load_layers(["config/base.yaml", path], env_vars)
    errors = sorted(jsonschema.Draft7Validator(schema).iter_errors(cfg), key=lambda e: list(e.absolute_path))
    ...

Its output on the three environments:

dev      ok    localhost:8080 tls=False pool=10 log=debug
prod     ok    app.example:8080 tls=True pool=40 log=warn
staging  FAIL  2 schema errors
           database/pool_size: '20' is not of type 'integer'
           server: Additional properties are not allowed ('tsl' was unexpected)

Staging has two mistakes that a parser accepts: the key tsl is a typo for tls, and the pool size is the string "20". Without the schema, staging would have started with TLS off and a string where a number was expected, and found out at the first connection. With it, the pull request that introduced them fails, with the path and the reason. The summary line for the passing environments shows the effective values, which doubles as a quick look at what each environment will run (validating config YAML with JSON Schema in CI pipelines).

One caution from testing. check-jsonschema loads YAML with 1.2-style rules, so a file with country: NO and ports [80:80, 22:22] passed as strings there while PyYAML loaded False and 1342. The project validates the data produced by the application's own loader, which sidesteps the problem. If you use another validator, quote ambiguous values or validate through the same loader.

Keep names and comments consistent

Pick one key style and write it down. Snake case for application settings is a good default, since it maps to code and to environment variable names. Enforce it with a schema that lists the known keys and forbids others, or with a propertyNames pattern for open-ended mappings; a test flagged serverHost, max-retries, DB and Port with their paths. Migrate old files with a recursive yq expression that inserts underscores and lowercases, and let the reader accept both spellings for a transition (naming keys consistently in YAML config files). Avoid keys that YAML 1.1 reads as booleans (on, no, yes).

Comments explain why, not what. A comment that says port: 5432 # PostgreSQL default adds nothing; one that says the vendor drops idle connections at 60 seconds prevents an outage. Temporary settings get an owner and a date. Lint the style with yamllint's comment rules, and remember that tools drop comments: anything a program needs is a key (comments that help the next person reading your YAML).

Reuse carefully

Inside one file, anchors and the merge key remove repetition. They are shallow: in a test, a service that defined its own environment replaced the anchored one entirely, so settings that must extend need to be restated. Keep anchors for a few lines repeated across entries, name them clearly and keep chains short (using anchors to reduce repetition in YAML config files). Across files, the layering above is the tool.

When one file grows past what a person can read, split it by seam (environment, subsystem, owner). Fragments merged in sorted order need only a merge step and stay plain YAML; a custom include tag needs your loader everywhere. Print the effective configuration with the file each value came from, since a split config otherwise turns every surprise into a search (splitting a large YAML config into several files).

A worked refactor: three copies to a base and overrides

The commonest starting point is three environment files that were once copies. Here are three, each with four sections (server, database, features, logging), forty lines in total:

server: {host: app.example, port: 8080, tls: true}
database: {url: "postgres://[email protected]/app", pool_size: 40, timeout: 60}
features: [search, billing]
logging: {level: warn, format: json}

(dev.yaml and staging.yaml differ in the host, TLS, pool size and log level.) The refactor extracts a base automatically. For each key, if a majority of the environments agree on the value, that value goes in the base, and each environment keeps only the outliers. A short script does it, comparing values as JSON so lists and nested mappings work, and recursing into mappings:

base    : {"server": {"port": 8080, "tls": true}, "database": {"url": "postgres://[email protected]/app", "pool_size": 10, "timeout": 30}, "features": ["search"], "logging": {"format": "json"}}
dev     : {"server": {"host": "localhost", "tls": false}, "logging": {"level": "debug"}}
staging : {"server": {"host": "staging.example"}, "logging": {"level": "info"}}
prod    : {"server": {"host": "app.example"}, "database": {"pool_size": 40, "timeout": 60}, "features": ["search", "billing"], "logging": {"level": "warn"}}
merged result equals original for every environment: True
settings lines before: 40 after: 30

The base took the port, TLS on (staging and production agree), the database URL, a pool size of 10 and timeout of 30 (dev and staging agree), the single-feature list and the log format. Development overrides TLS back to false and sets its host and log level; staging overrides only its host and log level; production overrides the host, the pool size, the timeout, the feature list and the level. The decisive line is the check: merging the base with each override reproduced the original file for all three environments, so behavior did not change. Forty lines became thirty, and the more important change is that each environment file now states what is different about that environment.

Two cautions. A majority base is a heuristic: it may make an unusual environment the default for the wrong reason, and a person should read the result and decide what the base ought to mean (here, "safe defaults" might mean TLS off in the base and on in staging and production). And the script treats lists as atomic, because the merge replaces lists; if your overlay tool appends lists, the equivalence check must use that tool's merge. Always verify with the real merge and the real loader, and keep the script's equality check as the gate: a refactor of configuration is correct when the effective configuration of every environment is byte-for-byte unchanged.

What belongs in the base

A setting belongs in the base when it is the same in most environments and a different value would need a reason. Defaults that express policy (log format, timeouts, pool sizes) belong there with a comment about why. Environment identity (host names, the environment name itself) belongs in the override, because it differs by definition, and the base should not carry a value that is wrong everywhere (host: localhost in the project above is a safe local default that the other environments override). Anything secret is a reference in the base and resolved from the environment.

A useful test: if the base file alone were deployed by accident, would it fail safe? A base with TLS off and debug logging deployed to production by mistake is a worse failure than a base with TLS on that fails to reach a dev database. Choose base values so that the dangerous direction needs an explicit override.

Adding a new setting

A short checklist keeps the structure healthy as the application grows. Add the key to the base with a default that is safe everywhere, or without a default and listed as required in the schema if every environment must choose. Add it to the schema with its type and constraints, and set additionalProperties: false in its section so a misspelling is caught. Write a comment with the unit, the reason and any link to other settings. Add the key to the example or documentation the team reads. Override it only in environments that differ. Run the check script, which validates each merged environment, and read the effective-configuration printout for the environment you changed. If the setting is renamed or removed later, follow the deprecation path rather than editing every file at once.

Ownership and review

Configuration needs owners like code does. Name the owning team in the header comment of each file, and use code-owner rules so that a change to production's override is reviewed by the people who run production. Review configuration changes for the questions in this guide: does it change the effective configuration of any environment other than the intended one, does it add a literal where a reference belongs, does it pass the schema, does the comment still match the value.

Keep the history useful. Commit formatting changes separately, write messages that say why a value changed, and link the incident or ticket. When an outage is traced to a configuration value, the commit that set it should explain the reasoning in a line, so the next person can change it with confidence or leave it alone for a reason. A configuration repository with a clear history, and a short README that says where the schema lives and how to run the checks, is the opposite of the file nobody dares to touch.

Version the format itself

The project's base has schema_version: 1, and the schema requires it. That field is easy to omit and expensive to lack. Configuration files outlive the code that first read them, and without a version the loader cannot tell a file written for an old layout from one written for the current layout. With it, a loader can refuse files it does not understand, migrate older ones and report precisely what it did.

Plan changes in three kinds. Compatible additions, an optional key with a default, need no version bump; add them to the schema as optional. Deprecations: keep accepting the old name for a period, log a warning that names the file and the key, and document the removal date. Breaking changes: bump the version, provide a migration script (a yq command or a Python function) and make the loader fail with the migration instructions if it sees the old version. The aim is that nobody finds out about a format change by their service failing to start.

A migration script is a test of your own design: run it over all the real files in the repository and validate the output against the new schema. If it cannot convert a file mechanically, the format change is bigger than you thought.

Testing the configuration, not only the files

Checks on files catch syntax and shape. A few tests on the effective configuration catch policy. Write them as plain tests in the application's test suite: load each environment through the real loader and assert the facts that must hold. Production has TLS on. Only development has debug logging. Every environment's pool size is at most the database's connection limit divided by the number of app servers. No environment's database URL contains a password. The set of feature flags in production is a subset of the flags the code knows about. These tests are short, they read like the policies they enforce, and they fail with a message that names the environment and the setting.

Add a test for the printout itself: the effective-configuration function runs without error for every environment and masks every field the schema marks as secret. And keep one test that loads the committed example local file, so the example cannot rot.

Tests like these also document intent for a new colleague better than a wiki page, because they are executable and cannot be out of date without failing. When a policy changes, the test changes in the same pull request as the configuration, and the review sees both. The team's decisions about configuration end up in the repository, next to the files they govern.

When the people who edit are not programmers

Some configuration is edited by operators, analysts or support staff. They should not need to know about schemas or merges. Give them what they need to succeed: a small file with a header comment that says what each section controls, an example they can copy, an editor setup that underlines mistakes, and a check they can run with one command that prints messages in plain language. The schema and loader exist to produce those messages.

Keep their file small. If the part they edit is thirty lines inside a three-hundred-line file, move it to its own file with its own schema, and let the application load it separately. Restrict what the schema allows to what they should change. And have a person on the engineering side review every change that reaches production, at least until the checks have earned trust. The goal is that a mistake by a non-programmer fails a check with an understandable message instead of failing a service, which is better for them and for you.

Operate it

Pin the tools. Versions of yamllint, check-jsonschema and yq in the CI image, printed in the job log. Behavior changes between releases, and one yq release printed a warning about changing its merge-anchor default.

Run the checks in three places: the editor, a pre-commit hook and CI, with the same commands and the same configuration files. An editor with a language server and the schema underlines mistakes as they happen.

Show the effective configuration. A --show-config flag that prints the merged result with sources and secrets masked answers most questions in a minute.

Review the diff for meaning. A normalized diff (yq -P 'sort_keys(..)') ignores layout and key order; a whitespace-ignoring diff does the opposite of what you want, because it reported no difference for a key that moved from inside a block to the top level (diffing YAML files without whitespace noise or reordering).

Log what you load. At start-up, one line: which files, which environment, the schema version, the number of overrides. When something is wrong in production, that line is the first thing you read.

Adopting this in an existing repository

You do not need all of it at once, and the order matters because each step makes the next easier.

1. Inventory. Run the audit: lint counts by rule, files with tabs or non-breaking spaces, files that fail a strict load (working with YAML from the terminal has the script). You learn how much is broken before you change anything.

2. Fix syntax and duplicates. These are bugs, and a CI check on them has no argument against it.

3. Add the strict loader and use it for every config read.

4. Write the schema from the current files. Start permissive, then forbid unknown keys once the known ones are listed.

5. Extract the base. Find the settings that are identical across environments, move them to a base and shrink the environment files to differences. Verify with the effective-configuration printout that nothing changed for any environment: the merged result before and after must be equal.

6. Move secrets out and rotate any that were committed.

7. Normalize layout with a formatter, in a commit of its own.

8. Add the checks to CI, as gates, one rule at a time.

9. Write the conventions in the repository: key style, list and null policy, the layer order, where to put new settings.

Step 5 deserves a test. Before changing anything, write a script that prints each environment's effective configuration as sorted JSON. After the refactor, run it again and diff. An empty diff proves the base extraction did not change behavior, and a non-empty one points at the exact setting that moved.

Failure modes of the structure itself

A good structure can still go wrong. Override files that grow until they repeat the base: review their size, and move repeated lines back. Too many layers: base, environment and local are enough; each extra layer multiplies the places to look. A schema that allows everything because someone relaxed it to make a build pass; make loosening a reviewed change. A loader that exists in two versions, one in the application and one in tooling, which disagree about duplicates or types; keep one module. Local files in the repository: a developer's debug override committed and shipped. Ignore them and commit only an example. And checks nobody runs: a CI job that is allowed to fail will fail permanently.

Related reading

The pieces: environment specific YAML config, keeping secrets out of config, splitting a large config, anchors, schema validation, comments and key naming. For the Python side read reading and writing YAML in Python safely, and for the syntax and errors YAML syntax you need to stop guessing at and fixing YAML errors that make no sense.

References: the twelve-factor app on config, the JSON Schema documentation, the yamllint documentation and the YAML 1.2.2 specification. Preview a layer in the YAML to JSON tool.

When this structure is more than you need

There is one environment and one maintainer. A single file with a few comments and a schema is enough. Add layers when a second environment appears.

The tool defines the structure. Kubernetes, Compose and CI systems have their own layout, validation and overlay mechanisms. Use them, and apply the habits here (secrets by reference, linting, review) around them.

The configuration changes at runtime. Frequently toggled flags belong in a flag service or a database with an audit trail, not in files that need a deploy.

You are tempted to build a framework. A merge function, a loader and a schema are enough. A configuration system with its own language and plugin model is a product, and you should buy or avoid it.

Your constraints differ from the tested ones. The project here is small, and the tools behave as their versions did on this machine. Larger systems with many services will want a generated schema, a shared loader package and an automated migration pipeline; the principles are the same.