Fixing Special Characters That Break YAML Values Safely

A YAML value is usually just text, until it contains a character that the grammar uses for something else. The failures come in three kinds: the parser refuses the file, the parser accepts it and loses part of the value, or the parser accepts it and changes characters inside it. The first is annoying, and the other two are the ones that reach production. This post tests the characters that cause each and gives the rule that covers them.

What the parser does with a plain value

An unquoted value, a plain scalar, runs to the end of the line, with two exceptions that end it early: a colon followed by a space (which starts a mapping) and a space followed by a hash (which starts a comment). The first character is also restricted: it cannot be one of the characters YAML uses as indicators. Everything else is allowed as is. The test results by group:

Characters that raise an error

t: {{ name }}      ->  ERR found unhashable key
n: - 1             ->  ERR sequence entries are not allowed here
k: !important      ->  ERR could not determine a constructor for the tag '!important'
msg: Error: disk   ->  ERR mapping values are not allowed here
token: @abc        ->  ERR found character '@' that cannot start any token

Braces. {{ name }} starts a flow mapping containing a flow mapping, and a mapping used as a key cannot be hashed, hence "found unhashable key". The message sounds like an internal error and means a template placeholder was left unquoted. Quoted, "{{ name }}" is the string {{ name }}. Templates such as Jinja and Helm use this syntax, so any YAML file that contains templating must quote its placeholders or be rendered before parsing.

A dash and a space. - 1 after a key is a list indicator in a place where one is not allowed. A negative number written without the space, -1, is fine and loads as an integer.

An exclamation mark. ! starts a tag. !important is read as a tag named important, which no loader knows. Quote the value.

A colon and a space inside a value and **a leading @** are covered in the error posts (mapping values are not allowed here and found character that cannot start any token).

Characters that silently truncate

A hash is only a comment marker when it follows whitespace:

u: http://x/a?b=c#frag      ->  {'u': 'http://x/a?b=c#frag'}
u: http://x/a?b=c #frag     ->  {'u': 'http://x/a?b=c'}

The first URL keeps its fragment. The second lost #frag, with no error, because the space turned the hash into a comment. This is dangerous because the result is plausible. A color #ff0000 after a key, a password containing #, a shell comment inside a value: all truncated. Passwords are the memorable case: password: abc #123 loads as abc, and the application fails to authenticate with a message that says nothing about YAML. Quote any value that might contain a space and a hash, and any secret.

Characters that change inside quotes

Double quotes process escape sequences, which makes them the wrong choice for text containing backslashes:

p: "C:\temp\new"        ->  'C:<TAB>emp<NEWLINE>ew'   (shown as 'C:\temp\new' by repr)
p: 'C:\temp\new'        ->  'C:\\temp\\new'            (backslashes kept)
p: C:\temp\new          ->  'C:\\temp\\new'            (plain, kept)
p: "C:\\temp\\new"      ->  'C:\\temp\\new'            (escaped backslashes)

The first line is the trap. In double quotes \t is a tab and \n is a newline, so the Windows path became "C:", a tab, "emp", a line break and "ew". Python's repr of the loaded string prints \t and \n for those characters, which looks identical to the path you wrote, so the bug is easy to miss in a console. The result is a path that does not exist. The fixes are single quotes, a plain scalar (which keeps backslashes) or doubled backslashes inside the double quotes. The same applies to regular expressions: pattern: "\d+" is an invalid escape in a double-quoted scalar and an error, while '\d+' is correct.

Single quotes have one special sequence: a doubled single quote means one. 'it''s' is it's. A value that contains both kinds of quotes and backslashes may be easier in a block scalar (YAML multiline strings with pipe and greater than).

Characters that are fine

Not everything special-looking needs quotes. These loaded as written, unquoted:

k: a,b        ->  'a,b'          (comma outside flow style)
k: [a,b]      ->  ['a', 'b']      (inside flow style a comma separates)
k: it's       ->  "it's"
k: say "hi"   ->  'say "hi"'
k: 50%        ->  '50%'           (percent not at the start)
n: -1         ->  -1

A comma matters only inside flow collections, so k: a,b is a string while k: [a,b] is a list. An apostrophe or double quote in the middle of a plain scalar is just a character. A percent sign is fine unless it starts the value. Over-quoting is not harmful, only noisy; the question is whether the cost of a mistake on a given value justifies it.

The rule

1. Default to single quotes for text with special characters. They are literal and have one rule to remember.

2. Use double quotes only when you need an escape (a newline, a tab, a Unicode escape), and double the backslashes otherwise.

3. **Quote anything that contains : , #, {{, or starts with @, ` `, %, !, *, &, [, {, |, >`, or a quote.**

4. Quote secrets and paths, where a silent change is expensive.

5. Use a block scalar for long or multi-line text.

6. Test it: load the file in a test and compare each tricky value with the expected string. A table of inputs and expected outputs, in code, documents the intent and catches the silent cases.

When a program writes the YAML, use a library: it applies these rules automatically, choosing quotes per value. String-building YAML with f-strings is where most of these bugs are born.

Related reading

The cluster hub is fixing YAML errors that make no sense. The wider quoting rules are in YAML strings, quoting and when you need quotes, and secrets in keeping secrets out of YAML config files. Escape sequences are listed in section 5.7 of the YAML 1.2.2 specification, and the plain scalar rules in section 7.3.3. See how a value loads in the YAML to JSON tool.

When quoting still leaves a value wrong

The consumer interprets the string again. A path or command that goes to a shell needs shell quoting on top. YAML quotes protect only the YAML layer.

A template engine runs first. If placeholders are replaced before parsing, a substituted value can contain : or a quote and break the document. Parse first, substitute after (environment variable substitution in YAML config).

The file passes through a formatter. A formatter may change quote style and re-escape characters. Compare the loaded values before and after.

The character is invisible. A non-breaking space or a zero-width character inside a value loads without an error and breaks comparisons later. Search for non-ASCII characters in files that should be plain.