Finding Tab Characters Hiding in a YAML File Quickly

The error message is exact: found character that cannot start any token, line 3, column 1. The line looks fine. It is indented, it is aligned with its neighbors and there is nothing visible in column 1. The character is a tab, and your editor has been showing it as spaces. This post is the toolkit for finding every tab in a file or a repository, replacing them safely and stopping them from returning. It also covers the invisible character that does not raise an error, which is worse.

Why you cannot see it

A tab character is one byte, 0x09. Editors render it by jumping to the next tab stop, so it looks like two, four or eight spaces depending on settings. Once rendered, it is indistinguishable from spaces. YAML forbids tabs for indentation, and the parser stops at the first one:

found character '\t' that cannot start any token | line 3, column 1

The position is exact; the problem is only that you cannot see what is at that position.

Find tabs in a file

grep with Perl-compatible patterns understands \t:

grep -nP '\t' t1.yaml

t1.yaml:3:	- run: build

-n prints the line number, -P enables the \t escape. The output shows the line with its tab (displayed as a wide gap). To make the tab visible, cat -A prints control characters in a readable form:

cat -A t1.yaml | sed -n 3p

^I- run: build$

^I is a tab and $ marks the end of the line, which also reveals trailing spaces. Use cat -A whenever a file behaves strangely: it shows tabs, trailing whitespace and Windows line endings (^M$) at once.

Find tabs in many files

For a directory tree, list the files that have a tab in their leading whitespace:

grep -rlP '^[ ]*\t' --include='*.yaml' --include='*.yml' .

./a.yaml

The pattern anchors at the start of a line, allows spaces, then a tab, which is what the YAML error is about (tabs after content are a different matter). In a Git repository git grep searches only tracked files, is fast and respects ignore rules:

git grep -nP '\t' -- '*.yaml' '*.yml'

a.yaml:3:	- run: build

Put this in CI as a check that fails when it finds anything. It costs a fraction of a second and gives the file and line.

Replace tabs

If a tab is used for indentation, replace it with spaces. The width is your choice, and two spaces is the usual:

sed -i 's/\t/  /g' a.yaml

After the replacement the file loads:

{'name': 'web', 'steps': [{'run': 'build'}]}

expand does the same with tab stops, which is what you want if the tabs were aligned to stops:

expand -t2 t1.yaml | cat -A | sed -n 3p

  - run: build$

On macOS the BSD sed -i needs an argument (sed -i '' ...) and does not understand \t in the pattern, so use expand or install GNU sed. Always check the result with a parse, not by eye, and look at the diff: a replacement that changes the number of leading spaces by even one can move a line to a different level. If the original tab was meant to be a single level, convert it to one level's worth of spaces and not to the visual width of a tab.

The invisible character that does not fail

The tab is the lucky case, because the parser rejects it. A non-breaking space (U+00A0), which looks exactly like a space, is not rejected. Two of them used as indentation:

{'a': None, '\xa0\xa0b': 1}

The parser read the second line as a key named with two non-breaking spaces and the letter b, at the top level. The structure silently changed: a is null and a strange key exists. Your program looks for b under a and finds nothing. These come from copying text out of web pages, chat tools and documents.

Search for the bytes (the UTF-8 encoding of U+00A0 is C2 A0):

grep -nP '\xc2\xa0' n.yaml | cat -A

2:M-BM- M-BM- b: 1$

cat -A prints each non-breaking space as M-BM- (the escaped bytes), so they stand out. A broader sweep finds any non-ASCII character in files that should be plain ASCII:

grep -nP '[^\x00-\x7F]' file.yaml

Review each hit. A legitimate accented letter in a string is fine; anything in indentation or around a colon is not. Replace non-breaking spaces with spaces:

sed -i 's/\xc2\xa0/ /g' file.yaml

Other invisible characters are the zero-width space (U+200B), which loaded in a test as part of a value, turning a mapping line into a single string, and the byte order mark at the start of a file, which PyYAML accepted without complaint. Use the same grep approach.

Keep them out

1. Editor. Insert spaces for the Tab key in YAML files, and show whitespace characters. An .editorconfig file with indent_style = space and indent_size = 2 for *.yml and *.yaml applies to most editors.

2. Linter. yamllint reports a tab as a syntax error with line and column, and its trailing-spaces and new-lines rules cover related whitespace (linting YAML with yamllint).

3. Pre-commit hook. A git grep -nP '\t' over staged YAML files, failing on a match, stops tabs at the source.

4. Paste carefully. Paste into a plain-text buffer first when copying from a web page or a document, or use "paste and match style" in tools that offer it.

5. CI. The same grep in CI catches what the hook misses.

Related reading

The cluster hub is fixing YAML errors that make no sense. The error itself is explained in YAML error: found character that cannot start any token, and the rules in YAML indentation rules and why tabs break everything. The tab restriction is stated in section 6.1 of the YAML 1.2.2 specification, and the editor config format at editorconfig.org. Reformat a fixed file in the YAML formatter.

When a character is invisible and not a tab

It is a non-breaking or zero-width space. The parser accepts it, so use the byte search above, not the YAML error.

The line ending is wrong. Windows line endings (\r\n) are accepted by YAML parsers, but a stray \r in the middle of a line is a character in the value. cat -A shows ^M.

The encoding is not UTF-8. A file saved in another encoding yields odd bytes. Check with file file.yaml and convert.

A formatter inserted the character. Rerun the formatter and diff: an automated tool that adds invisible characters is a bug to report.