Environment Specific YAML Config Without Copy and Paste

Staging, production, a developer's laptop. The same service, with different hosts, different limits and different secrets. The first solution is a copy of the config per environment, and it is the one that decays. Six months later the copies differ in ways nobody chose: a timeout fixed in production only, a feature flag left on in staging. This post describes the layered alternative, shows it working with real output, and covers the practices that keep it from becoming its own mess.

Why copies drift

Environment-specific configuration is mostly the same configuration. A service has thirty settings, and the environments differ in four of them. Copying the file thirty times means thirty settings are maintained in each copy, and every change to a shared setting has to be applied in several places by hand. People forget, and the copies diverge. The divergence is invisible: each file is valid, the service starts and the difference shows up as a behavior change that nobody can explain.

Overlays invert this. The shared settings live once, in a base. Each environment's file contains only what it changes. A change to a shared setting is one edit; a change to an environment is one edit in one small file.

The layers

server: {host: localhost, port: 8080}
features: [search]
logging: {level: info}

server: {host: app.example, tls: true}
logging: {level: warn}

logging: {level: debug}

The loader applies the base, then the environment file chosen by a variable, then an optional local file, each merged over the result so far. The production file does not mention the port, so the base's 8080 applies; the local file changes nothing but the log level. With production and local both applied the effective configuration is:

/server/host       = 'app.example' from 20-prod.yaml
/server/port       = 8080          from 10-base.yaml
/server/tls        = True          from 20-prod.yaml
/features          = ['search']    from 10-base.yaml
/logging/level     = 'debug'       from 30-local.yaml

The last column is the valuable part. It was produced by tracking, during the merge, which layer last set each leaf value. For a developer asking "why is this debug logging in my staging run?", the answer is on the screen: 30-local.yaml, a file that should not exist in staging. Build this printout into the application behind a flag (--show-config, or a debug endpoint restricted to operators) and mask secrets in it.

The merge function is the recursive one used elsewhere in this cluster: mappings combine, everything else is replaced (merging two YAML files in Python with a deep merge). On the command line, yq does the same:

yq eval-all '. as $item ireduce ({}; . * $item)' base.yaml "env/$ENVIRONMENT.yaml" > /tmp/effective.yaml

Rules that keep overlays healthy

Keep overrides small. If an environment file repeats most of the base, it is a copy with extra steps. Move the shared lines back to the base and leave only the differences. A rough check: an override file longer than a third of the base deserves a look.

Decide the list policy. A deep merge replaces lists. That is predictable but surprising for people who expect to add one item. If an environment needs the base list plus one more, either restate the whole list, or restructure as a mapping keyed by name, which merges by key. Write the policy at the top of the base file in a comment.

Decide what null means. An override that sets a key to null overwrites the base. Use that deliberately to unset, and lint for accidental empty keys (YAML null values and the empty string question).

One source of truth for the environment name. A single variable (APP_ENV) selects the file and is itself recorded in the effective configuration. Do not infer the environment from the host name in some places and a variable in others.

Secrets stay out of every layer. Overrides reference secrets by name like the base does (keeping secrets out of YAML config files).

Fail on an unknown environment. A typo in APP_ENV should stop the start-up with a list of valid names, not silently run with the base alone, which is the configuration of no environment at all.

Treat local files as untracked. A 30-local.yaml for developer overrides belongs in .gitignore, with a committed 30-local.example.yaml that shows the shape. Otherwise one developer's debug level ships to production.

Validating the result

Fragments are incomplete on purpose: the production file has no port. Validate the merged document, where every required key must be present, with a schema or a typed model that forbids unknown keys. That catches the typical overlay mistakes: a key misspelled in an override (which would be silently ignored without the check, since the base still provides the real one), a type mismatch (a string port) and a missing required value. Run the validation in CI for every environment, merging the real files, so a broken override fails before it is deployed (validating config YAML with JSON Schema).

A useful test is a table: for each environment name, load the layers and assert a few facts that must hold, such as "production has TLS on" and "no environment other than local has debug logging". They are cheap to write and document policy better than prose.

Overlays in other tools

Platforms have their own layering. Kubernetes uses Kustomize overlays and Helm values files, Docker Compose merges multiple -f files and CI systems have templates and variables. When the file belongs to such a tool, use its mechanism: its validators understand the result, and your merge function does not. What this post describes is for your own application's settings, where no platform decides for you.

Related reading

The cluster hub is structuring YAML configuration files that last. The mechanics of splitting are in splitting a large YAML config into several files, and the reuse inside one file in using anchors to reduce repetition in YAML config. The idea of environment-specific configuration kept apart from code comes from the twelve-factor app, and Kubernetes layering is described in the Kustomize documentation. Compare two layers in the YAML formatter.

When per-environment overlays are the wrong structure

The environments are very different. If production and development share little, overlays hide the difference. Separate files with a shared schema are clearer.

Environment settings change often at runtime. Feature flags toggled daily belong in a flag service, not in files that need a deploy.

The platform already layers. Use Helm values, Kustomize or the CI system's own mechanism.

Nobody can print the effective config. An overlay system without a way to show the result is a source of mysteries. Build the printout first.