Finding Duplicate Keys Across a Config Repository Reliably

Searching for "duplicate keys" in a repository of YAML is two different tasks wearing the same name. The first is a mistake inside a single file: the same key written twice at one level, where a parser quietly keeps one of the values. The second is a difference across files: the same key set to different values in base.yaml, prod.yaml and staging.yaml, which may be exactly what you want or a leftover from a copy-paste. Both are worth finding, they need different tools and they deserve different responses. This post runs both on a small repository. It belongs to the cluster on analyzing configuration and metadata held in YAML, and the underlying behavior is in YAML duplicate keys and silent overwrites.

Duplicates inside one file

A mapping with the same key twice is invalid according to the YAML specification and accepted by many parsers. The four environment files are:

name: app
port: 8080
db:
  host: db.dev.internal
  pool: 10
port: 8081
port: 8082

and staging.yaml, which sets log_level: debug and log_level: info on consecutive lines. What each tool says about the pair:

PyYAML loads them and keeps the last value:

PyYAML silently keeps: dev.yaml {'port': 8082}
PyYAML silently keeps: staging.yaml {'port': 8080, 'log_level': 'info'}

No error and no warning. The dev port is 8082 because it came last, and the values 8080 and 8081 are gone. The staging log level is info, and the debug line is dead text. A person who edits the file and adds log_level: debug at the top, not noticing the existing line further down, has no effect and no feedback. That is the scenario the checks exist for.

ruamel.yaml raises. A scan that loads every file with the round-trip loader and catches DuplicateKeyError finds them:

y = YAML(typ="rt")
for p in sorted(Path("repo").rglob("*.y*ml")):
    try:
        list(y.load_all(p.read_text()))
    except DuplicateKeyError as e:
        dupes.append((str(p), str(e).splitlines()[0][:70]))

files with duplicate keys within a file:
   repo/config/dev.yaml while constructing a mapping
   repo/config/staging.yaml while constructing a mapping

Exactly two files out of 35, found in one pass. load_all is used instead of load so multi-document files, such as the Kubernetes manifest in the same repository, are read in full. The exception's text continues past the first line with the key and its position, which you can print for a useful report.

yamllint has a rule for it, key-duplicates, enabled in the default configuration. Running it:

yamllint -d "{rules: {key-duplicates: enable}}" repo/config repo/helm repo/compose

repo/config/staging.yaml
  7:1       error    duplication of key "log_level" in mapping  (key-duplicates)

repo/config/dev.yaml
  6:1       error    duplication of key "port" in mapping  (key-duplicates)
  7:1       error    duplication of key "port" in mapping  (key-duplicates)

The linter reports the line of each duplicate occurrence. For dev.yaml, it flagged lines 6 and 7, the second and third port, and treated the first as the original. This is the form to use in a pre-commit hook and in CI (building a pre-commit check for YAML files), because it needs no code and names the line. The ruamel.yaml scan is useful when you want a custom report, or when the repository contains non-standard files that the linter's file patterns skip.

The response to a within-file duplicate is simple: fail the build. There is no legitimate use of a repeated key at one level, and the cost of fixing it is a deleted line. The harder question is which one the author meant, which the file's history answers.

Differences across files

Across the four environment files, a key set to different values is not an error; it is the point of having environment files. But unintended differences are drift, and they are worth listing. The method flattens each file to path and value pairs and groups by path:

vals = defaultdict(dict)
for p in sorted(Path("repo/config").glob("*.yaml")):
    d = yaml.safe_load(p.read_text())
    for k, v in walk(d):
        vals[k][p.stem] = v
for k, m in vals.items():
    if len(set(map(str, m.values()))) > 1:
        print(k, m)

with a walk helper that yields db.host-style dotted paths for the leaves (flattening nested YAML into rows). The report:

keys with different values across config files:
   port {'base': 8080, 'dev': 8082, 'prod': 8080, 'staging': 8080}
   db.host {'base': 'db.base.internal', 'dev': 'db.dev.internal', 'prod': 'db.prod.internal', 'staging': 'db.staging.internal'}
   log_level {'prod': 'warning', 'staging': 'info'}

Three keys differ. db.host differs everywhere, by design. port differs only in dev (8082, the result of the duplicate keys above), so it is probably a mistake and a candidate for removal from dev.yaml. log_level appears only in two files, so it is absent from base and dev: the report shows keys missing from some files because the dictionary lacks them. Whether base should define a default is a design question the report prompts.

The scan reads each file separately, so it reflects what each file says. If your application merges base and environment layers, compare the merged results instead, using the same merge function the application uses (testing config loading in CI). Then "different" means different in effect.

Noise control matters. A repository with many environment files and many keys produces a long list of differences, most intended. Filter in three ways: ignore keys known to differ (host names, secrets references), report only keys whose values differ in files that should agree (all production regions), and highlight overrides that equal the base value, which are redundant and hide the real overrides.

Both at once, and in other places

The two scans combine into one repository check: within-file duplicates fail the build, across-file differences print as a report. Add checks for duplicates in other shapes of data:

• Duplicate list entries, such as the same port or the same service name twice in a list. A uniqueness assertion after loading catches them; no YAML parser will.

• Duplicate anchors. Defining &defaults twice makes the second win for later aliases, and yamllint has an anchors rule with options for duplicates.

• Duplicate keys that differ in case or whitespace (Port and port, log_level and log-level). A normalizing pass that lowercases and replaces hyphens finds near-duplicates, which are typos waiting to happen. Naming keys consistently in YAML config covers enforcing a style.

• Duplicate keys across merged documents, where a key is defined in two included files and the order decides. Resolve the include order and report overrides.

Also remember the security angle: a parser that keeps the last duplicate key can be made to disagree with a validator that keeps the first. Rejecting duplicates at the door removes that class of bypass (YAML security pitfalls).

When a duplicate key search cannot find the real problem

The duplicates come from merging. A key defined in an anchor and again in the record is a feature (override), not a mistake. Do not flag merge overrides as duplicates.

The files are generated. If a template produces the duplicate, fix the template; editing the output is pointless.

The tool normalizes before you see it. A formatter that round-trips through PyYAML removes duplicates silently, so run the check on the file as committed.

The difference is meaningful but not textual. timeout: 30s and timeout: 30 mean the same thing in different units to different readers. Text comparison cannot see it.

The rule is documented in the yamllint rules reference, and the specification requirement in YAML 1.2.2 section 3.2.1.3. Check a file in the YAML formatter.