Merging YAML Files From the Command Line With yq Safely

Layered configuration is common: a base file with defaults, a second file with overrides for an environment. In Python a deep merge is a ten-line function. On the command line, in a CI step or a deployment script, yq does it in one expression. This post shows the expression, what each operator does to nested mappings and lists, and the traps around nulls, ordering and writing the result, tested on a base and an override file.

The two files

server:
  host: db1
  port: 5432
  opts: {ssl: true, pool: 5}
features: [search, export]

server:
  host: db2
  opts: {pool: 20}
features: [billing]

The override changes the host and the pool size and does not mention the port or ssl. It also supplies a different features list.

The deep merge

yq eval-all 'select(fi==0) * select(fi==1)' a.yaml b.yaml

server:
  host: db2
  port: 5432
  opts: {ssl: true, pool: 20}
features: [billing]

eval-all (alias ea) reads all the files into one stream of documents before running the expression, unlike plain eval, which runs the expression on each file separately. fi is the file index, starting at 0, so select(fi==0) is the first file and select(fi==1) the second. The * operator is the deep merge: where both sides have a mapping at the same key it recurses, and where they differ the right side wins. The result keeps port: 5432 and ssl: true from the base, takes host: db2 and pool: 20 from the override. A shallow merge would have kept only the override's server block.

The features list was replaced by the override's list, [billing]. That is the default behavior for arrays with *.

Appending lists

The variant *+ appends arrays:

yq eval-all 'select(fi==0) *+ select(fi==1)' a.yaml b.yaml

server:
  host: db2
  port: 5432
  opts: {ssl: true, pool: 20}
features: [search, export, billing]

Same mappings, and the list now holds all three. Other modifiers exist in the yq manual for specific needs: *d to merge arrays by index (deep), *n to only add keys that do not exist in the base (so the base wins), and *? to merge only keys that exist. Pick the one whose meaning matches your policy and write it in a comment next to the command. The point is the same as in the Python function: the list policy is a decision, and the operator is where it lives. Replacing is the usual choice for configuration where the override means "exactly these", and appending suits lists of things to run or paths to search.

Writing the result

yq prints the merged document to standard output, so redirect it to a new file:

yq eval-all 'select(fi==0) * select(fi==1)' base.yaml prod.yaml > merged.yaml

Do not use -i with several input files. In a test, yq -i '...' f1.yaml f2.yaml f3.yaml rewrote only the first file, replacing its content with the concatenation of all three as a multi-document stream, and left the other two unchanged (using yq to update many YAML files at once). For a merge, write to a new path, validate it and move it into place.

To merge more than two files, list them in order. Later files override earlier ones, and the expression reduces across them:

yq eval-all '. as $item ireduce ({}; . * $item)' base.yaml env.yaml local.yaml

The ireduce form folds a list of any length with the * operator. The simple select(fi==0) * select(fi==1) form works for exactly two files.

What the merge does to nulls and empties

A null in the override is a value like any other. If the override has debug: ~, the merged debug is null, replacing the base's false. Whether that is what you want is the same question as in Python: is a null "unset" or "no change"? yq has an operator to ignore nulls in some forms, and the simplest protection is to not write empty keys in override files. Lint for them with yamllint or a schema.

A key with an empty mapping (opts: {}) in the override merges as no change to the base's opts, since a deep merge of an empty mapping adds nothing, while a key with opts: ~ replaces it. The two look similar in a file and behave differently (YAML null values and the empty string question).

Comments and layout

yq keeps the comments of the inputs where it can, and the merged layout follows the first file with the second's new keys added at the end of their mappings. Flow-style mappings from the input (opts: {ssl: true, pool: 20}) stay in flow style. Run a formatter on the result if you commit it, so it matches the repository style. For merged files that are generated at deploy time and never committed, the layout does not matter.

Checks after merging

1. Load the merged file with the parser the application uses and print it. A merge can succeed and produce a value that the parser reads differently.

2. Validate against the schema. Unknown keys from a typo in the override pass through the merge untouched, and a schema with forbidden extras reports them (validating config YAML with JSON Schema).

3. Check for duplicate keys in each input first. A duplicate in the base is resolved by the parser before the merge, which hides it.

4. Diff the merged file against the base to review what the override actually changed: diff base.yaml merged.yaml.

5. Keep the layer order in the command, not in file naming alone, so it is visible in the script.

A deployment pattern

A common use is composing the effective configuration at deploy time. The repository holds base.yaml and one file per environment. The deploy script merges them into a temporary file, validates it, and mounts or uploads it:

yq eval-all '. as $item ireduce ({}; . * $item)' base.yaml "env/$ENVIRONMENT.yaml" > /tmp/effective.yaml
check-jsonschema --schemafile schema.json /tmp/effective.yaml

I ran the folding form on three files and the result combined all three layers: server took host: db2 from the second file and port: 1 from the third, with opts merged from the first two, and the extra key from the third appeared at the end. The effective file is an artifact, so do not commit it. Print it in the deploy log with secrets masked, because the first question in any incident is what configuration was actually applied.

Keep the base free of environment-specific values and the overrides small. An override file that repeats most of the base has stopped being an override, and the next change to the base has to be made in several places. The merge makes small overrides cheap, so use it that way.

Related reading

The cluster hub is working with YAML from the terminal. The same merge in code, with the policy discussion, is in merging two YAML files in Python with a deep merge, and layering by environment in environment specific YAML config without copy and paste. The operators are in the yq multiply and merge documentation, and the file-index variable in the yq fi reference. The merge semantics follow the YAML 1.2.2 mapping rules. Compare two inputs in the YAML formatter.

When a command-line merge is the wrong tool

Lists of records need to merge by key. Combining two lists of services by name is not what * or *+ does. Use a script, or yq's unique_by and group_by operators, with care.

The platform has its own overlay mechanism. Kustomize and Helm merge Kubernetes manifests with knowledge of the fields. Use them for those files.

The merge needs conditionals. Logic belongs in code with tests.

You must preserve exact formatting. Merge, then run the formatter your repository uses, or accept normalized output.