YAML Duplicate Keys and the Silent Overwrite Problem

The YAML specification says the keys of a mapping must be unique. Most parsers do not enforce it. PyYAML, the most widely used YAML library in Python, loads a file with a repeated key and quietly keeps the last value. The first one disappears without a message. In a configuration file, that means a setting that was reviewed, committed and deployed is replaced by another one lower in the file. This post shows what each tool does and how to make duplicates fail loudly.

What each tool does

The file:

a: 1
b: 2
a: 3

Results:

PyYAML default : {'a': 3, 'b': 2}
ruamel.yaml    : DuplicateKeyError  (found duplicate key "a" with value "3" (original value: "1"))
yamllint       : 3:1  error  duplication of key "a" in mapping  (key-duplicates)
yq '.'         : a: 1 / b: 2 / a: 3   (printed unchanged)

PyYAML reports nothing and returns a dictionary in which a is 3. The 1 is gone. ruamel.yaml raises an error that names the key and both values. yamllint reports it as an error with the line. yq read the file and printed it back with both entries, so a pipeline that merely passes the YAML through yq will carry the problem forward with no sign of it.

The difference between libraries is the reason this bug lives for a long time: whoever wrote the file used one tool, the application reads it with another, and no tool in between complains.

How duplicates get in

Merges. Two branches each add a setting to the same block, and Git merges them cleanly because the lines do not overlap. The result has the key twice. A textual merge has no idea that the file is a mapping (merging XML config files in Git without manual conflicts describes the same Git behavior for XML).

Copy and paste. A block pasted into a long file, with a key that already exists higher up.

Overrides. A person adds a key at the bottom meaning "this one wins", unaware that the key is already set above. It works, by accident, in PyYAML, and fails in a parser that rejects duplicates.

Anchors and merge keys. With <<: *base and a local key, the local key wins by design, and that is not a duplicate. Two local keys of the same name in one mapping are:

base: &b {x: 1, y: 2}
prod:
  <<: *b
  x: 9
  x: 10

merge + dup    : {'x': 10, 'y': 2}

PyYAML kept the last x, 10, and ignored the first, 9.

Nested blocks. The duplicate can be a whole block:

svc:
  env:
    A: 1
  env:
    B: 2

nested dup     : {'svc': {'env': {'B': 2}}}

The first env block, with A, vanished entirely. This is worse than a scalar override: the application loses a whole group of settings.

Make the loader strict

If you must use PyYAML, subclass the safe loader and replace the mapping constructor with one that checks for repeats:

import yaml
from yaml.constructor import ConstructorError

class Strict(yaml.SafeLoader):
    pass

def no_dupes(loader, node, deep=False):
    seen = set()
    for key_node, _ in node.value:
        key = loader.construct_object(key_node, deep=deep)
        if key in seen:
            raise ConstructorError(None, None, f"duplicate key {key!r}", key_node.start_mark)
        seen.add(key)
    return loader.construct_mapping(node, deep)

Strict.add_constructor(yaml.resolver.BaseResolver.DEFAULT_MAPPING_TAG, no_dupes)

Used with yaml.load(text, Loader=Strict), the three-line file fails with duplicate key 'a' at line 3. Note that the check applies to the merged mapping separately from merge keys: it looks at the keys written in this mapping, so a local override of an anchored key is allowed and two local copies are rejected. That is the behavior you want.

A stricter library saves the code: ruamel.yaml raises by default (you can allow duplicates with allow_duplicate_keys = True, which you should not). If you control the reader, switching removes the problem.

Catch it before it is read

Whichever loader the application uses, add the check to the pipeline:

• **yamllint** with the key-duplicates rule, enabled by default. Run it in CI and in a pre-commit hook. It covers the repository, regardless of which parser reads the files in production (linting YAML with yamllint).

• A merge check. After merging branches, run the linter on the result. A clean Git merge does not mean a valid file.

• A test that loads every config with the strict loader. Ten lines, and it catches the nested case too.

• Review. When a diff adds a key, search the file for it.

For generated YAML, a program that builds a dictionary cannot create duplicates, because Python dictionaries have unique keys. Duplicates come from text.

What to do when you find one

First decide which value is right, because the loader chose the last by accident of implementation, and the author may have meant the first. Look at the history (git log -p, git blame) to see who added each and why. Then delete the wrong one and keep a comment on the survivor if the reason is not obvious. If both are right because they are separate blocks (two env groups), merge them into one block by hand.

Finally, check that no other file has the same mistake. Duplicates are a pattern: if one slipped through, the checks that would have caught it are missing, and others will have slipped through too.

A note on order and meaning

When a loader keeps the last duplicate, the order of the lines becomes meaningful, even though the specification says a mapping has no order. People discover this and use it: a defaults block at the top and overrides at the bottom, all in one mapping. It works in PyYAML and in a few other libraries, and it is a fragile contract. A different parser may keep the first, may raise, or may keep both in a multimap. The same file then means different things in different tools, which is exactly the situation the uniqueness rule exists to prevent.

If you want defaults and overrides, say so with the tools YAML gives you: anchors and merge keys for inheritance, separate files for layering, or an explicit structure such as defaults: and overrides: that your program merges. Each of these is visible to a reader and survives a change of parser. A duplicate key, by contrast, looks like an accident to the next person who reads it, and often is.

Finally, treat a duplicate found in a deployed file as an incident worth a short write-up. It means a setting has been taking a value that nobody chose on purpose, possibly for a long time. Find out which value was live, whether anything depended on it and whether the other copy would have behaved differently, and record the answer next to the fix.

Related reading

The cluster hub is fixing YAML errors that make no sense. For the surrounding structure rules, see YAML lists and mappings without confusion, and for merges YAML anchors, aliases and merge keys. The uniqueness rule is stated in section 3.2.1.1 of the YAML 1.2.2 specification, and the lint rule in the yamllint reference. Check a file in the YAML formatter.

When duplicate keys are not the real problem

The repeated name is intentional in a different context. The same key in two different mappings is normal. Only repeats within one mapping are a problem.

The data should be a list. If you find yourself repeating a key to express several values, such as several ports entries, the structure should be a list under one key.

The tool uses multiple documents. A stream with three dashes between blocks has one mapping per document, and a key repeated across documents is fine.

The file is not meant to be a mapping. Some formats use repeated keys as a sequence of directives. That is a different format, with its own parser.