Environment Variable Substitution in YAML Config Files

Configuration files need values that differ per environment and that should not be in version control: a database password, a host name, a port. The common answer is placeholders in the file, ${DB_PASS}, filled from the environment when the file is loaded. The way it is filled matters more than it seems. Substituting into the text before parsing is the obvious way and the wrong one. This post shows the failure with a real password, the correct order, and the details of a small implementation.

The file

db:
  password: ${DB_PASS}
  host: ${DB_HOST:-localhost}
  port: ${DB_PORT}

The syntax follows shell: ${NAME} takes the variable, ${NAME:-default} takes the default when the variable is unset. The environment: DB_PASS is p@ss: word # 1, DB_HOST is unset, DB_PORT is unset at first.

The wrong order: expand, then parse

The simplest implementation applies os.path.expandvars to the file's text and then parses the result:

expanded = os.path.expandvars(raw)
yaml.safe_load(expanded)

expand-then-parse: ScannerError - mapping values are not allowed here

After expansion the line is password: p@ss: word # 1. The YAML parser sees a colon followed by a space inside the value and fails. Even when it does not fail, a value with # is truncated at the hash, and a value with a newline injects new lines into the document. The environment variable is now part of the YAML text, and anyone who controls the variable controls the structure of the file: a value of x\nadmin: true adds a key. That is injection, the same class of bug as building SQL by string concatenation. The text approach also has a second failure: expandvars leaves ${DB_HOST:-localhost} unexpanded, since it does not know the shell default syntax, so the placeholder stays in the config as a literal string.

The right order: parse, then expand

Load the YAML first. The placeholders are ordinary strings in the data. Walk the structure and replace placeholders in string values only:

import os, re, yaml

PAT = re.compile(r"\$\{(\w+)(?::-([^}]*))?\}")

def sub(s):
    def one(m):
        name, default = m.group(1), m.group(2)
        if name in os.environ:
            return os.environ[name]
        if default is not None:
            return default
        raise KeyError(f"environment variable {name} is not set")
    return PAT.sub(one, s)

def expand(node):
    if isinstance(node, dict):
        return {k: expand(v) for k, v in node.items()}
    if isinstance(node, list):
        return [expand(v) for v in node]
    return sub(node) if isinstance(node, str) else node

With DB_PASS set and DB_PORT not:

parse-then-expand: KeyError 'environment variable DB_PORT is not set'

After setting DB_PORT=5432:

parse-then-expand: {'db': {'password': 'p@ss: word # 1', 'host': 'localhost', 'port': '5432'}}

The password is exactly as set, colon and hash included, because it never went through the YAML grammar: it was inserted into an already-parsed string. The host took its default, localhost. The missing port raised a KeyError that names the variable, which is a far better failure than a config with an unexpanded placeholder. The structure of the document cannot be changed by the environment, because only string values are touched.

Details of the implementation

Types. Substituted values are strings: the port is '5432', not 5432. Convert after expansion, in the validation step: a pydantic model declared port: int converts a numeric string, and fails on abc with a clear message (validating YAML configuration with pydantic). Do not try to guess types during substitution.

Partial substitution. The pattern replaces placeholders inside longer strings, so url: postgres://app:${DB_PASS}@db/app works. If the value must be percent-encoded for a URL, do that in code; the password in this test contains a colon and a hash that would break a URL.

Required versus optional. A placeholder without a default is required, and a missing variable raises. A default makes it optional. Make the choice visible in the file.

Escaping. If a value legitimately contains ${, you need an escape such as $${ that the function turns into a literal. Decide whether you need it before you meet the case.

Keys. The walk above substitutes in values only. Substituting in keys is rarely wanted and opens the door to surprises; leave keys alone.

Where to find the variables. os.environ is global. For tests, pass a mapping to the function and default to the environment. That makes the function pure and easy to test.

Logging. Never log the expanded configuration. It contains the secrets. Log the keys, or log with secret values masked.

A flow-style trap

Placeholders contain braces and sometimes a colon, and both have meaning inside a YAML flow mapping. Writing a placeholder in flow style fails to parse:

'a: {h: ${X:-y}}'      -> ERR expected ',' or '}', but got '{'
'a: {h: "${X:-y}"}'    -> {'a': {'h': '${X:-y}'}}
'a:\n  h: ${X:-y}'     -> {'a': {'h': '${X:-y}'}}

In block style the placeholder is an ordinary plain scalar, because braces in the middle of a plain scalar are allowed. Inside braces or brackets the parser reads the placeholder's own brace as the start of another mapping and stops. The fix is to quote the placeholder in flow context, or to use block style for any line that carries one. Lint for it, since the error message points at the placeholder and looks unrelated to substitution.

Alternatives

Some systems solve this differently. Docker Compose substitutes variables itself and has its own default syntax (${VAR:-default}), applied to the raw text by the tool, which is why Compose files with awkward values need quoting. Kubernetes uses Secret objects and references them instead of substituting into manifests. Twelve-factor style applications skip the file for secrets entirely and read environment variables in code, with the YAML holding only non-secret defaults. Libraries such as pydantic-settings read environment variables into typed models directly, with a file layer underneath.

The pattern that scales best is: the file declares which variable supplies each secret (password_env: DB_PASS), and code reads the environment. The file then contains no secret and no template syntax, and the YAML stays plain data (keeping secrets out of YAML config files).

Testing it

Write tests for the awkward values, not the easy ones: a secret with a colon, a space and a hash; a value containing a quote and a newline; an unset variable with and without a default; an empty variable (set but empty), which name in os.environ treats as set, so the empty string wins over the default. Decide whether you want that. The shell's ${VAR:-default} treats empty as unset, while ${VAR-default} does not, so choose which one your syntax follows and document it.

Related reading

The cluster hub is reading and writing YAML in Python safely. The loading step is in loading YAML in Python with PyYAML safe_load, and layering files in merging two YAML files in Python with a deep merge. The shell syntax is described in the POSIX parameter expansion section, and Compose's rules in the Docker documentation. Inspect a parsed file in the YAML to JSON tool.

When variable substitution should not be done in YAML

The value is a real secret. Prefer a secret store or the platform's mechanism, and have the file refer to it by name.

The file is consumed by another tool. If Compose, Helm or an orchestrator reads the file, it applies its own substitution, and yours would double-process the placeholders.

You need complex logic. Conditionals and loops in YAML are a path to unreadable templates. Generate the configuration from code instead.

The environment is untrusted. If an attacker can set variables, substitution is a channel for them. Validate the result against a schema and forbid extra keys.