Keeping Secrets Out of YAML Config Files and Git History

Every team that keeps configuration in YAML eventually finds a password in it. It was meant to be temporary, or it was a development credential, or somebody pasted the production value to debug. Removing it from the file is the obvious fix and the wrong place to stop. This post shows what happens to a secret in Git, how to structure the configuration so it never has secrets in it, and how to scan for the ones that got through. The examples use a made-up password in a throwaway repository.

What happens to a secret in Git

A repository with a password in config.yaml, committed, then removed in a second commit:

db:
  host: db.internal
  password_env: DB_PASSWORD

The working file is clean. Now ask Git who ever saw the old value:

git log -p --all -S hunter2 --oneline

701c093 move password to environment
-  password: hunter2
c9572c7 add config
+  password: hunter2

Both commits appear. The first added the secret and the second removed it, and the removal is itself a record of the value (the minus line). -S searches for commits that change the number of occurrences of a string, which finds both. Anyone who cloned the repository before the fix has the secret. So does every fork, every CI cache that checked out the old commit and every backup. Deleting the line in a new commit changes the latest version only.

The consequence is a rule, not a technique: a secret that was committed is compromised. Rotate it. Rewriting history (git filter-repo, or the BFG tool) removes the value from the repository you control and is worth doing, and it does not recall the copies that already exist. Rotation is the fix; history rewriting is cleanup.

The pattern: reference, don't store

The configuration should name the secret and never contain it. Two forms work:

db:
  host: db.internal
  password_env: DB_PASSWORD        # name of the environment variable

db:
  host: db.internal
  password: ${DB_PASSWORD}         # placeholder expanded after parsing

In the first, the application reads the variable named by password_env. In the second, a loader replaces ${DB_PASSWORD} in string values after parsing. Do the expansion after parsing, because expanding the raw text first breaks on passwords with a colon and a hash, and lets the environment change the structure of the file (environment variable substitution in YAML config files tests this with a password containing both). Either form keeps the YAML shareable: it can be committed, reviewed, pasted into a ticket and posted to a forum without harm.

Where does the value live? In the environment of the process (set by the deployment platform from its secret store), in a secrets manager the application queries at start-up, or in a file mounted at run time, outside the repository, that the application reads. For Kubernetes, Secret objects referenced from the pod. For encrypted-in-Git workflows, tools such as SOPS keep an encrypted file in the repository and decrypt at deploy time; I did not test one here, so check their documentation for the key-management model before relying on it.

Make the loader enforce the pattern. Fail with a clear message when a required variable is missing, and treat a literal value in a field that should be a reference as an error:

def require_ref(cfg):
    pw = cfg["db"].get("password")
    if pw and not (isinstance(pw, str) and pw.startswith("${")):
        raise ValueError("db.password must be a ${VAR} reference, not a literal")

Scanning for secrets that got through

Scan two places: the current tree and the history.

git grep -nEi '(password|secret|token|api_key)[a-z_]*:\s*[^$ ]' -- '*.yaml'

config.yaml:3:  password_env: DB_PASSWORD

The pattern looks for a key name with a secret-like word followed by a value that does not start with $. It matched password_env: DB_PASSWORD, a false positive: the value is the name of a variable, not a secret. Every simple pattern has these. Allowlist the keys that hold references (*_env), or make the pattern skip them, and keep the pattern list in a file under version control so it can be refined. A real scanner (gitleaks, trufflehog, GitHub's secret scanning) uses many patterns and entropy checks and has maintained allowlists; use one, and treat the grep as a quick check.

The history scan is the one people forget:

git log -p --all | grep -nEi '^\+.*(password|secret|token|api_key)[a-z_]*:\s*[^$ ]'

15:+  password_env: DB_PASSWORD
31:+  password: hunter2

The second hit, the added line with hunter2, is the finding: a secret in history, and the old value that the working tree no longer shows. Run history scans when you take over a repository, when you open-source one and on a schedule. Anything found is rotated first and cleaned up second.

Add the scan where it cannot be skipped: a pre-commit hook that scans the staged diff and blocks the commit, and a CI job that scans each pull request and the default branch. The hook catches the mistake before it enters history, which is the only point at which it costs nothing.

Other places secrets hide in YAML

Comments. A password left in a comment is as exposed as one in a value. The scan patterns above only look at keys; widen them to comment lines.

URLs with credentials. url: postgres://app:[email protected]/app carries a password in the userinfo part. A separate pattern (://[^/@ ]+:[^/@ ]+@) found it in the test file. Store the URL without credentials and add them at run time.

Example and test files. Copies of production configuration used as fixtures, with real values.

Generated and rendered files. The output of a template, committed by accident, with the secret expanded.

Logs. A start-up message that prints the loaded configuration. Mask secrets in log output (a SecretStr type in the validation model prints as **********).

Docker and CI settings. Image build arguments and pipeline variables that embed values in image layers or logs.

A short policy

1. Configuration files contain no secrets. They contain references.

2. Secrets come from the environment or a secret store at run time.

3. A scanner runs in the pre-commit hook and in CI, on the diff and on history at intervals.

4. A secret found in Git is rotated the same day, then removed from history.

5. Logs never print secret values.

6. Example files use fake values that cannot be mistaken for real ones.

Related reading

The cluster hub is structuring YAML configuration files that last. The substitution mechanics are in environment variable substitution in YAML config files, and validation that rejects literal secrets in validating YAML configuration with pydantic. The history tools are documented in the git-filter-repo project and GitHub's guidance on removing sensitive data. Check a file in the YAML formatter.

When removing a secret from a file is not enough

It was ever pushed. Assume it is public. Rotate.

It is in a comment, a URL or a fixture. Search for those forms.

The scanner reports a false positive. Allowlist it explicitly with a reason, so the scanner stays trusted.

The secret has a long life. Long-lived credentials are the problem. Prefer short-lived tokens issued by the platform, which turn a leak into a short window.