Merging Two YAML Files in Python With a Deep Merge

You have a base configuration and an override: defaults plus an environment's changes, or a shared file plus a local one. Combining them looks like {**base, **override}. It is wrong in the way that matters most: it replaces whole blocks. This post shows the shallow merge fail, writes the recursive merge in ten lines, and covers the decisions a merge function must make about lists and nulls, because the answers are policy, not code.

The two files

# base.yaml
server: {host: db1, port: 5432, opts: {ssl: true, pool: 5}}
features: [search, export]
debug: false

# override.yaml
server: {host: db2, opts: {pool: 20}}
features: [billing]
debug: ~

The override changes the host and the pool size and leaves the port and ssl alone (by not mentioning them). It also replaces the features list and sets debug to null. The intended result keeps the port and TLS setting.

The shallow merge

{**base, **override}

shallow update: {'server': {'host': 'db2', 'opts': {'pool': 20}}, 'features': ['billing'], 'debug': None}

The server key from the override replaced the base's server completely. The port 5432 and ssl: true are gone, though the override never mentioned them. dict.update and the ** merge operate on the top level only. For flat configuration it is fine. For nested configuration it silently drops settings, which is how a service ends up running on a default port in production.

The deep merge

A recursive function merges mappings key by key and replaces everything else:

import copy

def merge(x, y, lists="replace"):
    out = copy.deepcopy(x)
    for k, v in y.items():
        if isinstance(v, dict) and isinstance(out.get(k), dict):
            out[k] = merge(out[k], v, lists)
        elif isinstance(v, list) and isinstance(out.get(k), list) and lists == "append":
            out[k] = out[k] + v
        else:
            out[k] = copy.deepcopy(v)
    return out

replace lists: {'server': {'host': 'db2', 'port': 5432, 'opts': {'ssl': True, 'pool': 20}}, 'features': ['billing'], 'debug': None}
append lists : {'server': {'host': 'db2', 'port': 5432, 'opts': {'ssl': True, 'pool': 20}}, 'features': ['search', 'export', 'billing'], 'debug': None}

Now server has the override's host (db2), the base's port (5432), and an opts mapping that combines both: ssl from the base, pool 20 from the override. The function recurses where both sides are mappings and takes the override's value otherwise. The deepcopy calls matter: without them the result shares nested dictionaries with the inputs, and a later change to the merged config would change the base too. This is the same sharing behavior that makes YAML aliases surprising (YAML anchors, aliases and merge keys).

Decisions the code cannot make for you

Lists: replace or append. The two outputs above differ on features. Replace says the override's list is the whole list. Append says it adds to the base's. Both are legitimate and documented in different tools: configuration layering usually replaces, because "I want exactly these features" is the common intent, while lists of things to run or paths to search sometimes append. A third policy, merging lists of mappings by a key (name), is what Kubernetes strategic merge does and needs a key per list. Choose one, write it in the documentation, and put it in the function name or an argument. Do not mix policies silently.

Null in the override. debug: ~ overwrote the base's false with None. That is correct if null means explicitly unset. It is a surprise if the author wrote a key with no value as a placeholder and expected "no change" (YAML null values and the empty string question). A common rule: null removes the key, so you can unset a base setting. Another: null is ignored, so a placeholder is harmless. Pick one. Whichever it is, an empty value after a colon is easy to write by accident, so lint for it.

Type conflicts. If the base has a mapping and the override has a scalar for the same key, the override wins in the function above. Often that is a mistake in the override. Decide whether to raise.

Unknown keys. A key in the override that is absent from the base may be a typo. A validation step after the merge catches it (validating YAML configuration with pydantic).

Order. Python dictionaries keep insertion order, so the merged dictionary has the base's keys first and then new keys from the override. If order matters in the output file, say so.

Merging with more than two files

Layer by folding: functools.reduce(merge, [base, env, local]), or a loop over files sorted by name (10-base.yaml, 20-prod.yaml, 30-local.yaml), each merged onto the result so far. The sorted-by-name directory pattern is easy to reason about and common in tools. In the example in this cluster, the effective config after each file made it clear which file set which value, and printing the intermediate results is a cheap debugging aid (splitting a large YAML config into several files).

Keep the origin of each value if you can. A configuration object that records which file set each leaf lets you answer "why is the port 8080?" without guessing. It costs a parallel structure, and it saves hours.

Preserving comments

If the merged result is written back to a file that people read, a plain dictionary merge loses comments. Do the merge on ruamel.yaml round-trip objects, which behave like dictionaries; the function works unchanged on them, and comments attached to the base keys survive (preserving comments and order with ruamel yaml). Test that the merge keeps the type of container: the function above copies with deepcopy, and the result is the same class as the input.

The command-line equivalent

yq has a deep-merge operator, *, and an array-appending variant, *+:

yq eval-all 'select(fi==0) * select(fi==1)' base.yaml override.yaml

server:
  host: db2
  port: 5432
  opts: {ssl: true, pool: 20}
features: [billing]

and with *+ the list became [search, export, billing], the same two policies as the Python function. The command is useful in shell scripts and CI (merging YAML files from the command line).

Related reading

The cluster hub is reading and writing YAML in Python safely. Anchors and merge keys inside one file are in YAML anchors, aliases and merge keys, and environment layering in environment specific YAML config without copy and paste. Python's copy semantics are in the copy module documentation, and Kubernetes' list-merge strategies in its strategic merge patch docs. Merge two snippets visually in the YAML formatter.

When a generic deep merge is the wrong tool

Lists of records have identity. Merging two lists of services by position is wrong. Merge by a key such as name.

The structure is a schema, not data. For Kubernetes-style objects, use the platform's own patch or overlay tool, which knows the fields.

You need to remove things. A merge only adds and replaces. Removal needs an explicit convention, like null meaning delete.

The inputs are untrusted. A merge of attacker-controlled data into trusted defaults can set keys you did not intend. Validate the result against a schema with forbidden extras.