Why JSON Parse Fails on a Byte Order Mark and How to Fix It

The JSON in the file is valid. You can see it. It opens in an editor, a formatter accepts the pasted text, and the same file parses on the colleague's machine. On yours, the parser fails at the very first character and the message points at something you cannot see. The usual cause is a byte order mark: three bytes at the front of the file that most editors hide. This article explains what they are, why strict parsers reject them while some tolerate them, what each language prints, and how to read, remove and prevent the mark.

What a byte order mark is

A byte order mark (BOM) is the Unicode character U+FEFF placed at the start of a text file. In UTF-16 and UTF-32 it tells the reader which byte order the file uses, which is its original purpose. UTF-8 has no byte order to announce, but Microsoft tools historically wrote it anyway as a signature saying "this file is UTF-8", and it encodes as the three bytes EF BB BF. Notepad did this for years, PowerShell's Out-File and the > redirect in Windows PowerShell 5 write UTF-16 or UTF-8 with a BOM depending on options, and Excel's "CSV UTF-8" save adds one (a cause of the same trouble covered in the UTF-8 BOM in CSV files).

JSON's standard has a position on this. RFC 8259, section 8.1, says that implementations MUST NOT add a byte order mark to the beginning of a JSON text, and that parsers MAY ignore the presence of one rather than treating it as an error. So a writer that adds a BOM is breaking the specification, and a parser that rejects it is within it. Both statements are true at once, which is why the same file works in one tool and fails in another.

What the parsers do

I wrote the file with the three bytes in front and read it five ways in Python 3.13:

json.loads(raw_bytes)                          -> {'a': 1}
json.loads(raw_bytes.decode("utf-8"))          -> JSONDecodeError: Unexpected UTF-8 BOM (decode using utf-8-sig): line 1 column 1 (char 0)
json.load(open(path, encoding="utf-8"))        -> JSONDecodeError: Unexpected UTF-8 BOM (decode using utf-8-sig): line 1 column 1 (char 0)
json.load(open(path, encoding="utf-8-sig"))    -> {'a': 1}
json.loads(raw_bytes.decode("utf-8-sig"))      -> {'a': 1}

Two things stand out. When you hand json.loads bytes, it detects the encoding itself (it looks for UTF-8, UTF-16 and UTF-32 signatures) and handles the BOM. When you hand it a string that was decoded as plain UTF-8, the BOM has become the character U+FEFF, and the parser refuses it with a message that gives the fix. Python's message is the most helpful of the group. For a UTF-16 file, json.loads on the bytes worked as well, and opening the file with encoding="utf-16" also worked, because that codec consumes its BOM.

Node is less friendly:

JSON.parse("{\"a\":1}") -> SyntaxError: Unexpected token '', "{"a":1}" is not valid JSON

The message quotes the offending character, which prints as nothing, and the document it quotes looks fine. If you copy that message into a ticket, the BOM may not even survive. Browsers' fetch(...).json() strips a leading BOM in UTF-8 decoding according to the Fetch standard, but JSON.parse on a string read by other means does not. PHP's json_decode returned an error (Syntax error) on the same text, and jq 1.6 accepted a file with a BOM. A tool-by-tool inventory is the reason to normalize input yourself and not rely on whatever the parser forgives.

Confirming it is a BOM

Look at the first bytes, not the characters:

head -c 3 data.json | xxd

Output of ef bb bf confirms it. In Python, open(path, "rb").read(3) gives b'\xef\xbb\xbf', and in a string, text[0] == "" or repr(text[:1]) shows ''. file data.json on Linux and macOS prints "UTF-8 Unicode (with BOM) text". The same check catches the other invisible start, a UTF-16 file: ff fe or fe ff as the first two bytes.

When the message says "line 1 column 1" and the document looks right, check for a BOM before anything else. It takes a second and it is the cause often enough to deserve the first look. The wider list of what to check at position zero is in unexpected token in JSON at position 0.

Reading the file anyway

Pick the fix that matches your language and leaves the file untouched.

Python. Open with encoding="utf-8-sig". That codec strips a BOM if one is there and behaves like plain UTF-8 if not, so it is safe for files from both kinds of writer:

with open("data.json", encoding="utf-8-sig") as f:
    data = json.load(f)

For a response from requests, use json.loads(r.content) with bytes, or r.content.decode("utf-8-sig"). Do not use r.text if the charset guess is wrong, because requests may fall back to ISO-8859-1 for a text/* type with no charset and give you mojibake as well.

JavaScript. Remove the character before parsing: JSON.parse(text.replace(/^/, "")). In Node, fs.readFileSync(path, "utf8") leaves the BOM in the string. A tiny helper that strips it belongs next to every file read.

Shell. sed '1s/^\xEF\xBB\xBF//' data.json > clean.json removes it from a copy (on macOS the stock sed does not understand \x escapes, and a Perl one-liner, perl -pe 's/^\xEF\xBB\xBF//' data.json, does). For many files, a loop with the same command. Always write to a new file first and compare.

Databases and loaders. PostgreSQL's COPY treats a BOM as part of the first value, so a JSON column loaded from a file with one fails at the first row in the same way. DuckDB's read_json tolerated a UTF-8 BOM in my earlier tests on the parser edge cases (see what JSON allows and what parsers quietly accept), but do not depend on it.

Stopping it at the source

A reader that strips the BOM is a workaround. The files come from somewhere, and fixing that stops it for every consumer:

• Editors. Save as "UTF-8" and not "UTF-8 with BOM". In VS Code the setting is files.encoding, in the status bar you can reopen or save with a chosen encoding. Notepad on current Windows 10 and 11 defaults to UTF-8 without a BOM, older builds did not.

• PowerShell. Windows PowerShell 5.1's Out-File and > produce UTF-16 LE with a BOM by default and -Encoding utf8 adds a BOM; PowerShell 7 writes UTF-8 without one. For a file that other tools will read, use PowerShell 7, or write with [System.IO.File]::WriteAllText($path, $text, (New-Object System.Text.UTF8Encoding $false)).

• Exports from Excel and some reporting tools. They add BOMs for the sake of Excel itself. If a tool offers "UTF-8" and "UTF-8 with BOM", pick the first for JSON.

• Your own code. In .NET, new UTF8Encoding(true) writes a BOM and new UTF8Encoding(false) does not, and Encoding.UTF8 writes one in some APIs. Java's OutputStreamWriter does not. Check once and add a test that reads back the first three bytes.

What else sits at the start of the file

Not every invisible character is a BOM, and the repair differs. A zero-width space (U+200B) looks the same but is a different character, and json.loads reports it as Expecting value and not the BOM message. Whitespace that JSON allows (space, tab, line feed, carriage return) is fine. A form feed or a vertical tab is not. If stripping a BOM does not fix the file, print repr(text[:20]) and name every character, as described in fixing JSON copied from a web page with smart quotes.

When the BOM is not the real problem

Treating a BOM as the cause is right when the failure is at position 0 and the first bytes are EF BB BF. It misleads you in these cases:

The file is UTF-16. A UTF-16 file saved by Windows tools begins with FF FE, and every other byte is 00. Reading it as UTF-8 yields a stream of null characters. The fix is the right encoding, not BOM removal. Python's json.loads(bytes) handled it in my test, while open(path) with the default encoding did not.

Another prefix is present. An XSSI guard such as )]}', or a log timestamp in front of the document is a prefix to strip on purpose, and a BOM remover will not touch it.

The BOM is in the middle. Concatenating two files that each began with a BOM leaves one in the middle of the output, which is an "Extra data" or "Expecting value" error at an inner position, not at the start. Strip the BOM from each file before joining.

The BOM is a symptom of mixed tooling. If a team's files keep getting BOMs, add an .editorconfig with charset = utf-8 and a CI check that fails on EF BB BF. Cleaning on read alone lets the problem spread to every new consumer.

For the rule itself, see RFC 8259, section 8.1, and for the codec behavior the Python documentation on the utf-8-sig codec.