Round Tripping YAML Without Losing Comments or Order

Reading a YAML file, changing one value and writing it back sounds like the simplest edit in the world. Do it with the most common Python library and the file comes back with its comments gone, its shared definitions copied out and its quotes stripped. The data is the same and the document is not, and in a repository where people read the file, the document is the point. This post runs one small file through three tools and shows what each preserves. It is part of the cluster on converting YAML to and from JSON, CSV, XML and Excel, and the Python mechanics are introduced in preserving comments and order with ruamel.yaml.

The test file

The file has a header comment, an end-of-line comment, a sequence indented flush with its parent, an anchor with a merge key and a single-quoted string:

# service settings
service:
  name: orders   # keep in sync with the repo
  port: 8080
  tags:
  - web
  - public
base: &base
  retries: 3
use:
  <<: *base
  retries: 5
quote: 'single'

The edit is one value: change port to 9090. A perfect round trip leaves every other byte alone.

PyYAML: the data survives, the document does not

import yaml

data = yaml.safe_load(open("rt.yaml"))
data["service"]["port"] = 9090
print(yaml.safe_dump(data, sort_keys=False))

With the port edit, the output was:

service:
  name: orders
  port: 9090
  tags:
  - web
  - public
base:
  retries: 3
use:
  retries: 5
quote: single

Everything you care about besides the value is different.

• Both comments are gone. # service settings and # keep in sync with the repo are not part of the data, so the loader discarded them.

• The anchor and merge key are gone. &base and <<: *base were resolved on load. use now holds only retries: 5, because the merge key was flattened and the override replaced the inherited retries: 3 (the original use mapping still had the same effective value, so nothing is wrong with the data, but the sharing is lost).

• The quotes are gone. 'single' became single. For this word that is harmless. For 'yes' or '02134', the dumper adds quotes back because it knows the rule, so the quotes you see are not your quotes.

If the file is machine-written, none of this matters. If humans maintain it, this is a destructive edit, and the pull request will show a diff far beyond the one value.

ruamel.yaml: the round-trip loader

ruamel.yaml keeps the document model, not just the data. Its default mode is round-trip:

import io
from ruamel.yaml import YAML

y = YAML()
y.preserve_quotes = True
data = y.load(open("rt.yaml"))
data["service"]["port"] = 9090
buf = io.StringIO()
y.dump(data, buf)

The comparison that matters:

identical except port: True

buf.getvalue() == src.replace("8080", "9090") was true, so the output matched the input to the byte, including the three spaces before the comment, the anchor, the merge key and the quoted string. The preserve_quotes = True line is needed; without it the single quotes are normalized away. That is what you want for an automated edit of a hand-maintained file.

The loaded objects are subclasses of dict and list (CommentedMap, CommentedSeq) that carry the comment and layout information, so most code works unchanged. Convert to plain types only at the edge, if a library demands it.

Indentation is a setting, not a property of the file

ruamel.yaml can write what it read, and it also has an idea of how YAML should look. The indent() method sets it. Setting mapping=2, sequence=4, offset=2 indents list items two spaces under their parent key, which is the style many teams use:

y.indent(mapping=2, sequence=4, offset=2)

On our file the sequence changed from flush to indented:

  tags:
    - web
    - public

The data is the same, and the diff now includes two lines nobody edited. The rule: set indent() to match the file you are editing, not the style you prefer. If the file mixes styles, pick the dominant one and expect to touch the others, or do not use a parser to edit it. Tools that reformat on commit make this moot, since the file is normalized once and stays that way; the YAML formatter is one way to do that normalization by hand.

yq: comments kept, whitespace normalized

yq is convenient for edits in scripts, and it keeps comments and anchors. Changing the port with yq '.service.port = 9090' rt.yaml and diffing against the original showed:

3,4c3,4
<   name: orders   # keep in sync with the repo
<   port: 8080
---
>   name: orders # keep in sync with the repo
>   port: 9090
6,7c6,7
<   - web
<   - public
---
>     - web
>     - public

The comment is there, but the three spaces before it became one, and the sequence moved two spaces right. The anchor and merge key and the quote were untouched, since the diff does not list them. Appending a list entry with yq -i '.service.tags += ["internal"]' produced the same re-indentation plus the new line. So yq is safe for comments and structure and not byte-exact. For a repository where diffs are reviewed, that is usually acceptable if you run the edit once and commit the normalization separately from the change. The in-place and multi-file mechanics are in editing a YAML value in place with yq and updating many YAML files with yq.

When the diff must be one line: patch the text

None of the parsers promise a minimal diff in every case. If a script must change one scalar in a file that people maintain, treat it as a text edit with a safeguard:

1. Find the line with a pattern that is as specific as you can make it (^ port: 8080$ under the right parent).

2. Replace it.

3. Load the result with a real parser and check that the value changed and nothing else did, comparing the loaded data of before and after.

Step three is what keeps the approach honest: a regular expression edit is fast and fragile, and the data comparison is the net. A script that finds more than one match, or none, should stop. This is the approach for lockfiles, version pins and CI files where a format-normalizing edit would cause churn.

Choosing

When a round trip cannot keep your YAML intact

The file uses features the library does not model. Rarely used constructs such as complex keys, unusual tags or non-default line breaks can change on a round trip. Test with your own file and diff the result with no edit applied: the no-op round trip is the best test.

Different line endings or encodings. A file with CRLF line endings or a byte order mark may be normalized on write. Check git diff for whitespace changes before committing.

The comments describe values you changed. A comment that says # port for the staging proxy above a value you now point elsewhere is preserved and wrong. No tool fixes that.

Several writers edit the same file. Parser-based edits reflow, and reflows conflict in merges. Agree on a single formatter and run it in CI.

The no-op round trip is the cheap test to run before trusting any tool with a file. The ruamel.yaml behavior is described in its documentation, and the YAML rules for comments are in the YAML 1.2.2 specification. For the PyYAML side, see writing YAML from Python.