Working With YAML From the Terminal Using yq and Friends

A lot of YAML work happens in a terminal: checking what a manifest says, changing an image tag in forty files, merging a base and an override in a deploy script, converting a file to JSON for another tool. The tools for it are few and good, and every one has a trap that the documentation mentions in one line. This guide is a tested tour of the tools, grouped by what you want to do, with the output of each command and the traps found in testing. The detailed posts are linked from the sections.

The toolbox

**yq** is the workhorse. It queries with a path language, edits in place, merges, converts between YAML, JSON, CSV, XML and other formats, and keeps comments. Two different programs share the name: the Go yq by Mike Farah, used in this guide, and a Python wrapper around jq. Their syntax differs, so check which you have with yq --version and pin it in scripts.

**yamllint** checks syntax and style and exits nonzero on errors. It is the CI check.

**check-jsonschema** validates a YAML file against a JSON Schema and prints every violation with its path. It is the shape check.

**diff, grep, sed, git** do the plain-text jobs: finding tabs, comparing normalized files, replacing characters, tracking changes.

**python3 -c** is the escape hatch: python3 -c "import yaml,sys,json; print(json.dumps(yaml.safe_load(open(sys.argv[1])), indent=1, default=str))" file.yaml prints the structure as the application's own parser sees it, which settles arguments about types.

The principle that organizes everything below: the terminal tools are for looking and for mechanical edits; anything with logic belongs in a script with tests.

Looking: query

A Compose file with three services is the sample. The commands and their output:

$ yq '.services.web.image' compose.yaml
nginx:1.25
$ yq '.services[].image' compose.yaml
nginx:1.25
postgres:16
redis:7
$ yq '.services | keys' -o=json -I=0 compose.yaml
["web","db","cache"]
$ yq '.services | to_entries | .[] | select(.value.image == "redis:7") | .key' compose.yaml
cache
$ yq '[.services[] | select(has("environment") | not)] | length' compose.yaml
1
$ yq '[.. | select(tag == "!!str" and test("^postgres"))]' -o=json -I=0 compose.yaml
["postgres:16"]

A dotted path reads a value, [] iterates, keys lists, select filters, .. searches everywhere and tag tests the parser's type. The trap to remember: a missing key prints null and exits 0, so a script holds the string null. Add -e to fail on absence (querying YAML with yq expressions and real examples). When you need to know how the parser typed a value, yq '.v | tag' on v: 3.10 printed !!float, a quick check for version numbers that will lose their zero.

Reading from a script

Shell makes quoting hard, and the data you read can contain anything. A value it's "quoted" $HOME date` came through yq -r untouched, and -o=shell produced note='it'"'"'s "quoted" $HOME date', a safely quoted assignment. The rules are the shell's: always double-quote expansions, loop with while read -r and never eval text from a file you do not control. A missing key needs -e (Error: no matches found, exit status 1 against null` and 0 without it) (reading YAML from a shell script without breaking quotes).

Editing

yq -i edits in place:

$ yq -i '.services.web.environment.MODE = "staging" | .services.cache.image = "redis:8"' compose.yaml
$ diff compose.yaml.orig compose.yaml
5c5
<     image: nginx:1.25   # pinned on purpose
---
>     image: nginx:1.25 # pinned on purpose
9c9
<       MODE: prod
---
>       MODE: staging
15c15
<     image: redis:7
---
>     image: redis:8

Exactly the two requested changes, plus one unrequested: the spacing before a trailing comment was normalized. Comments themselves are kept, which PyYAML would not have done. Appending a list item shows a deeper trap. yq -i '.services.web.ports += ["22:22"]' wrote - 22:22 without quotes, and PyYAML loaded the ports as ['80:80', 1342], because a 1.1 parser reads 22:22 as base 60. Forcing the style (.services.web.ports[] style="double") kept it a string (editing a YAML value in place with yq without damage). An assignment to a path that does not exist creates it, so a typo (.service.web.image) adds a stray key and exits 0. Read the diff after every edit.

Editing many files

The tempting form is wrong:

$ yq -i '.env.REGION = "eu"' svc1.yaml svc2.yaml svc3.yaml
 M svc1.yaml          # only the first file changed
svc1.yaml now has 29 lines and 2 document separators; svc2 unchanged: yes

-i updates the first file only, writing the combined output of all inputs into it as a multi-document stream. Loop instead, with one call per file. The first scripted pass over six files also normalized layout, 30 insertions and 24 deletions, and the real edit was six lines. Commit the formatting pass alone, then the edit, and the second diff is six inserted lines you can review in seconds. Loop inside one container when using Docker: six separate runs took 1.2 seconds and one container 0.2 (using yq to update many YAML files at once).

Merging

yq eval-all loads every file, and select(fi==0) picks by file index. The * operator deep-merges mappings, and *+ also appends lists:

$ yq eval-all 'select(fi==0) * select(fi==1)' a.yaml b.yaml
server:
  host: db2
  port: 5432
  opts: {ssl: true, pool: 20}
features: [billing]
$ yq eval-all 'select(fi==0) *+ select(fi==1)' a.yaml b.yaml
...
features: [search, export, billing]

The base's port and ssl survived, the override's host and pool applied, and the list was replaced or extended depending on the operator. That choice is the list policy, and it belongs in a comment beside the command. For three or more files, . as $item ireduce ({}; . * $item) folds them in order. Write the result to a new file, never with -i over several inputs (merging YAML files from the command line).

Converting

yq -o=json '.' file.yaml converts to JSON, -I=0 makes it compact, -p=json -o=yaml goes back, and a multi-document stream becomes JSON lines. The converter decides the types: on one file yq printed "country": "NO" and "flag": "yes" as strings and 3.10 as written, where PyYAML produced false, true and 3.1. A twenty-digit integer made yq stop with value out of range. Going the other way, yq wrote the string NO without quotes, which a 1.1 reader loads as false (converting YAML to JSON with yq and back again). A few other conversions come up often:

$ yq -o=csv '.rows' rows.yaml
name,v
a,1
b,2
$ yq -s '"split/" + .metadata.name' stack.yaml && ls split
one.yml  two.yml
$ yq 'select(document_index == 1) | .kind' stack.yaml
B

CSV from a list of records, one file per document from a stream and a single document picked by index.

Checking style and shape

yamllint reports problems with a line, column, level and rule name:

messy.yaml
  3:11      error    too many spaces after colon  (colons)
  5:5       error    wrong indentation: expected 2 but found 4  (indentation)
  6:8       error    too many spaces after hyphen  (hyphens)
  ...
exit=1

Nine findings on a deliberately messy file; errors give exit status 1 and warnings do not. The rules that catch bugs are syntax, key-duplicates and truthy; the rest are style, and line-length and document-start are the ones teams relax (linting YAML with yamllint). Lint finds style, not meaning. For meaning, validate against a schema:

$ check-jsonschema --schemafile schema.json bad.yaml
Schema validation errors were encountered.
  bad.yaml::$: Additional properties are not allowed ('retries' was unexpected)
  bad.yaml::$.env: 'production' is not one of ['dev', 'staging', 'prod']
  bad.yaml::$.server: Additional properties are not allowed ('tsl' was unexpected)
  bad.yaml::$.server.port: '443' is not of type 'integer'
  bad.yaml::$.features: ['search', 'search'] has non-unique elements

Five errors in one run, each with a path. It parses YAML 1.2 style, so env: NO was reported as the string 'NO' not being one of the allowed values, not as a boolean (validating config YAML with JSON Schema).

Comparing

Do not review YAML with diff -w. On two files where c was nested under a in one and at the top level in the other, diff -w printed nothing and exited 0, because the only difference is leading whitespace, and in YAML leading whitespace is structure. Normalize instead:

$ diff <(yq -P 'sort_keys(..)' old.yaml) <(yq -P 'sort_keys(..)' new.yaml) && echo semantically identical
semantically identical

The two files had different key order, flow versus block style and spacing. The normalized diff ignores those and still reports the nesting change. sort_keys leaves list order alone, as it should. Add ... comments="" to ignore comments (diffing YAML files without whitespace noise or reordering).

A CI job that uses all of it

The checks compose into a short pipeline. In a CI configuration:

yaml-checks:
  steps:
    - run: yamllint -c .yamllint --strict .
    - run: check-jsonschema --schemafile schemas/app.json config/*.yaml
    - run: |
        for f in config/*.yaml; do
          yq -e '.env' "$f" > /dev/null || { echo "$f: missing env" >&2; exit 1; }
        done
    - run: git diff --exit-code   # no formatter should have changed anything

Lint for style and bugs, validate shape, spot-check required keys with -e, and confirm that a formatting pass produces no diff, which proves the committed files are already normalized. Pin the versions of all three tools in the image so that a new release does not change the answers.

How yq thinks

Knowing a few ideas about the model makes the expressions predictable. yq parses the file into a tree of nodes, and every node carries more than its value. It has a tag (the type the parser gave it, !!str, !!int, !!float), a style (plain, single-quoted, double-quoted, literal or folded, block or flow), and comments (head, line and foot). Expressions read and write those properties as well as values. That is how style="double" forces quotes, how ... comments="" strips comments, and how a trailing comment survives an edit: it is attached to the node.

An expression is a pipeline. The current node is ., a path like .a.b descends, | feeds the result of the left side into the right, and operators such as select, map, to_entries and keys transform streams of nodes. Because [] and .. produce several results, a pipeline can produce many outputs; yq prints each in turn. Wrapping an expression in square brackets collects the stream into a list, which is why [.services[] | select(...)] | length counts matches: the brackets turn a stream into one list that length can measure.

Assignments are expressions too. = assigns the value on the right to every node selected on the left, evaluating the right side against the document root. |= updates each selected node using its own old value, evaluating the right side against that node, so .replicas |= . + 1 adds one to the existing number. The distinction decides whether . means "the document" or "this node", and mixing them up is the commonest reason an assignment does nothing or sets the wrong thing.

With several files, eval (the default) runs the expression on each file separately, and eval-all loads all of them as a stream, which is what merges and cross-file comparisons need, selected by file index with fi. Within one file, a multi-document stream is processed document by document, selectable with document_index. Understanding that eval is per file and eval-all is global explains most multi-file surprises, including the -i problem: in-place update targets the first file, so the other inputs' output has nowhere else to go.

Recipes beyond the basics

A few operations come up often enough to keep at hand. Each ran against a small test file.

Rename a key by copying and deleting: .services.web.env = .services.web.environment | del(.services.web.environment) produced a web block with env: where environment: had been, comments intact.

Delete list items by condition: del(.items[] | select(.name == "old")) removed the one record named old from a three-item list and left the other two untouched, in flow style as written.

Keep a list unique after appending: .tags += ["b"] | .tags |= unique on tags: [a, b] left [a, b], instead of a list with b twice.

Turn records into CSV: yq -o=csv '.rows' on a list of two records printed a header row name,v and two data rows, ready for a spreadsheet.

Split a stream into files named from their content: yq -s '"split/" + .metadata.name' stack.yaml wrote one.yml and two.yml, one per document. Pick one document by position with select(document_index == 1), and count by kind with yq -N '.kind' stack.yaml | sort | uniq -c.

Pass a value from the environment safely with strenv(NAME): yq -n '.region = strenv(REGION)' printed region: eu when REGION=eu was set. Interpolating a shell variable into the expression string instead lets a value with a quote break the expression, or change it. If yq runs in Docker, pass the variable into the container with -e REGION; without it the expression saw an empty value and produced region: "".

Real tasks from start to finish

Bump an image tag everywhere. List the files that mention the image with grep -l 'registry.example/svc' services/*.yaml, run the formatting pass and commit it, then loop yq -i '(.. | select(tag == "!!str" and test("^registry.example/svc:")) ) |= sub(":[0-9.]+$", ":2.0")' over the files, read git diff --stat and load every file with the production parser. The recursive search finds the image wherever it sits, including in init containers and sidecars that a fixed path would miss.

Audit what is deployed. yq '.services[].image' *.yaml | sort | uniq -c | sort -rn lists every image and how often it is used across a set of Compose files. For Kubernetes manifests the recursive form, yq '.. | select(has("image")) | .image', finds images in any workload kind. Pipe the result into a vulnerability scanner or a report.

Compare two environments. Normalize both with yq -P 'sort_keys(..)' and diff them. The output is the list of settings that differ, with layout and key order out of the way. Use it before a release to see what is not the same between staging and production.

Check required keys before deploying. A loop of yq -e '.path' file > /dev/null calls over the required paths fails the job at the first missing one. For a large set of rules use a schema and check-jsonschema, which reports all violations at once.

Build a report. Extract fields from many files into CSV with yq -o=csv or with yq -r '[.name, .replicas] | @csv'-style expressions, sort them and open the result in a spreadsheet. The output is text you can review, share and commit.

Each of these is a few lines. When one grows past ten lines of shell, or needs error handling for files that vary, move it to a script in a language with a YAML library and a typed model. The terminal tools are fast for the first draft, and the draft often shows what the script should check.

Installing and pinning

Install yq as a single static binary from the project's releases, from your system's package manager, or run it in a container. The binary is the best choice for CI and for scripts that call it many times, since container startup dominates the runtime for small files, as the six-container run showed. The container is the best choice when you cannot install software and can run Docker; mount the working directory and pass environment variables explicitly.

Pin the version in the place where it is installed, and print it in the job log. Behavior changes between releases are real: the run in this guide printed a warning that the handling of merge anchors was scheduled to change its default, which means a file with anchors and merge keys can convert differently after an upgrade. Check which yq you have before running a command you copied: the Python wrapper of the same name takes jq syntax and different flags, and a script that works with one fails confusingly with the other. A one-line guard at the top of a script (yq --version | grep -q mikefarah) prevents a wasted hour.

Document the tools a repository expects in its contributing notes: the versions of yq, yamllint and check-jsonschema, the config files they read and the single command that runs all the checks. New contributors then run the same checks as CI, and the checks stop being a surprise at review time.

A repository audit in one script

The tools above are strongest when combined. Here is a small audit script that you can run on an inherited repository to learn what state its YAML is in, before you decide what to enforce. It counts lint findings by rule, lists files with tabs or non-breaking spaces, and tries to load every file with a strict loader that rejects duplicate keys:

#!/usr/bin/env bash
# usage: audit.sh dir
dir=${1:-.}
files=$(find "$dir" -name '*.yaml' -o -name '*.yml' | sort)
total=$(echo "$files" | wc -l)
echo "files: $total"
echo "-- syntax / style (yamllint, parsable)"
/tmp/jx/bin/yamllint -d relaxed -f parsable $files 2>/dev/null | sed -E 's/.*\(([a-z-]+)\)$/\1/' | sort | uniq -c | sort -rn | sed 's/^/   /'
echo "-- files with tabs in indentation"
grep -lP '^[ ]*\t' $files 2>/dev/null | sed 's/^/   /'
echo "-- files with non-breaking spaces"
grep -lP '\xc2\xa0' $files 2>/dev/null | sed 's/^/   /'
echo "-- files that do not parse under the strict loader"
/tmp/jx/bin/python - $files <<'P'
import sys, yaml
from yaml.constructor import ConstructorError
class S(yaml.SafeLoader): pass
def nd(l, n, deep=False):
    seen = set()
    for k, _ in n.value:
        key = l.construct_object(k, deep=deep)
        if key in seen: raise ConstructorError(None, None, f"duplicate key {key!r}", k.start_mark)
        seen.add(key)
    return l.construct_mapping(n, deep)
S.add_constructor(yaml.resolver.BaseResolver.DEFAULT_MAPPING_TAG, nd)
bad = 0
for f in sys.argv[1:]:
    try:
        list(yaml.load_all(open(f), Loader=S))
    except yaml.YAMLError as e:
        bad += 1; m = getattr(e, "problem_mark", None)
        print(f"   {f}: {e.problem}" + (f" (line {m.line+1})" if m else ""))
print(f"   {bad} of {len(sys.argv)-1} files failed")
P

Run on a folder of 16 deliberately broken files:

files: 16
-- syntax / style (yamllint, parsable)
        13 syntax
         1 key-duplicates
         1 indentation
-- files with tabs in indentation
   .../bad/t1.yaml
   .../bad/t2.yaml
-- files with non-breaking spaces
-- files that do not parse under the strict loader
   .../bad/d1.yaml: duplicate key 'a' (line 3)
   .../bad/e1.yaml: mapping values are not allowed here (line 2)
   ...
   15 of 16 files failed

The report answers the questions that decide how to proceed. Thirteen of fifteen lint findings were syntax errors, so a first CI check that fails only on syntax would have caught nearly everything. Two files had tabs, which the strict loader also reported (with the same message the parser gives, which is why tabs count in both lists). The duplicate key was caught by both yamllint and the strict loader. The sixteenth file loaded fine. Use the numbers to pick the order of work: fix syntax, then duplicates, then tabs, then style.

The script is a starting point. Add the checks that matter for your files: a schema validation, a search for unquoted strings that look like booleans or numbers, a search for secrets. Keep it in the repository and run it in CI as a report, and later as a gate. Its value is that it makes the state of the YAML visible, and visible problems get fixed.

Terminals, editors and CI

The same checks should run in three places, and they should be the same checks. In the terminal, you run them by hand while you work. In the editor, a language server or a linter plugin runs yamllint and a schema check as you type, so the faults are underlined before you save. In CI, the pipeline runs the complete set on every pull request, with pinned tool versions, and fails the build on errors. When all three call the same commands with the same configuration files, a green editor means a green pipeline, and a surprise at review time becomes rare.

Keep the configuration in the repository: .yamllint, the schema files, an .editorconfig with the indentation and whitespace rules, and a short script or Makefile target (make yaml-check) that runs everything. New contributors run one command and get the same answer as CI. When a rule changes, one commit changes the editor, the terminal and the pipeline together.

Finally, treat the tools' output as a conversation with the file, not a verdict. A finding is a prompt to look: sometimes the file is wrong, sometimes the rule is wrong for this file and the right response is a documented exception. The goal is not zero warnings for its own sake. It is files that mean what their authors think they mean, which is a smaller and more useful goal.

What the terminal cannot tell you

Every command in this guide inspects text or a parsed tree. None can tell you whether the configuration is right for the system that reads it. A port that is already in use, an image tag that does not exist, a host name that does not resolve and a feature flag that the application ignores are all valid YAML and valid against a permissive schema. The checks here narrow the space of failures to the ones that only the system can reveal.

Close that gap with checks against the real thing, run as late as is safe. A dry run of the deployment tool, which parses and validates the manifest against its own schema without applying it. A container start with the configuration in a throwaway environment. A smoke test that calls the service after a rollout. These belong to the deployment pipeline and not to the YAML tooling, and they are the reason it is sensible to keep the YAML checks fast and cheap: they are the first filter, and the later stages are the last.

It also helps to keep a record of what the checks found over time. If the same class of mistake appears three times in a quarter, a rule or a schema change should follow, so that the fourth is caught by a machine. A short log of incidents next to the configuration, with the fix and the check that now prevents it, turns scattered lessons into a policy. The tools do the checking, and the people decide what is worth checking, which is the part no command-line utility can do.

Choosing between yq and Python

Both can do almost everything in this guide, so the choice is about the shape of the task. Use yq when the job is a single expression and the audience reads shell: reading a value for a deployment script, bumping a version across files, converting a format for a pipe. The expression sits in the script, it is short, and anyone who can read a command line can review it.

Use Python when the job has branches, needs data from elsewhere, produces structured error messages or must be tested. A script can load every file with a strict loader, validate it with a typed model, merge layers with a stated policy and print every problem with a path and a line number. It costs more lines and gives you a unit test, a debugger and the full set of language tools. The crossover is usually around the third pipe in an expression or the second if in a shell script.

Many teams settle on both. Short yq commands in Makefiles and CI steps for mechanical work, and one Python module that every application import uses for loading, so that the rules about duplicates, types and secrets live in a single place (reading and writing YAML in Python safely). Whichever you choose for a task, write down why, so the next person does not replace a working shell line with a framework or a working module with a pile of pipes.

Habits that prevent most problems

1. Pin and identify the tool. yq --version and a version in the install step.

2. Read every diff. Side effects of tools are small and constant: spacing, quotes, moved comments.

3. Edit a copy, or a clean branch.

4. **Loop over files; never -i with several.**

5. Commit formatting separately from content.

6. Load the result with the consumer's parser. It settles type questions.

7. **Use -e where absence matters.**

8. Quote ambiguous values ("22:22", "NO", "3.10") so no tool or version can reinterpret them.

9. Keep logic out of the shell. When an expression passes three pipes or a script passes thirty lines, write it in Python with a typed model.

Related reading

The posts in this cluster take one job each: querying, editing in place, merging, converting, linting, reading in shell scripts, bulk updates and diffing. For code instead of commands, see reading and writing YAML in Python safely, and for the syntax underneath YAML syntax you need to stop guessing at.

The tools: the yq manual, the yamllint documentation, check-jsonschema and the jq manual, whose language yq borrows. Inspect a file in the YAML to JSON tool or the YAML formatter.

When the terminal is the wrong place for YAML

The change needs logic. Conditionals, lookups against other data and per-file rules belong in a script that has tests.

The files are templates. Tools that parse YAML cannot handle placeholders. Operate on the values files, or on the rendered output.

Many people edit the same files by hand. Layout churn from tools creates conflicts. Agree on one formatter and run it in CI.

You need exact YAML 1.1 behavior. yq follows its library's typing, and PyYAML-based consumers follow another. Test the consumer.

The versions differ from these. Flags and warnings change between yq releases; the output above came with a warning about merge-anchor behavior that was scheduled to change. Re-run the commands on your version before building on them.