Fixing JSON Copied From a Web Page With Smart Quotes

You copy a JSON example from a documentation page, a Slack message, a wiki or a Word document, paste it into a file, and the parser refuses it. The text looks identical to a document that worked yesterday. The cause is not in the structure. Somewhere between the author and your clipboard, a program that thinks it is helping with typography replaced a plain character with a similar-looking one. This article shows how to see those characters, how the parsers report them, a repair that does not damage the legitimate text, and how to stop it happening again.

Characters that look right and are not

Word processors, chat applications, blogging platforms and some wikis apply "smart" typography: straight quotes become curly, three dots become an ellipsis, a double hyphen becomes a dash, and some insert non-breaking spaces to keep words together. Documentation sites sometimes do it to code samples that were typed in an ordinary text field. The seven characters I meet most often in broken JSON are these:

U+201C LEFT DOUBLE QUOTATION MARK
U+201D RIGHT DOUBLE QUOTATION MARK
U+2018 LEFT SINGLE QUOTATION MARK
U+2019 RIGHT SINGLE QUOTATION MARK
U+00A0 NO-BREAK SPACE
U+200B ZERO WIDTH SPACE
U+FEFF ZERO WIDTH NO-BREAK SPACE

The first four replace " and '. The last three are spaces or invisible marks in the places where a plain space or nothing should be. JSON allows exactly four whitespace characters between tokens: space, tab, line feed and carriage return. A non-breaking space is none of them, so to the parser it is garbage.

What the parsers print

Take {“name”: “Ada”, “age”: 36}, where the quotes are U+201C and U+201D. Python 3.13 says:

JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)

Node 24 says Expected property name or '}' in JSON at position 1 (line 1 column 2). This is the same message as for ordinary single quotes, which is why this error is confusing: the text on screen has quotes in every place a quote belongs. If you have already checked for single quotes and found none, the curly quote is the next suspect. The messages for the other characters are different.

A no-break space before a value, as in {"a": followed by U+00A0 and 1}, fails with Expecting value: line 1 column 6 (char 5) in Python. The position is the space itself, which is an honest answer but not a useful one, because the space looks like column 6 of a normal document. Node printed Unexpected token ' ', "{"a": 1}" is not valid JSON for the same input. Look at the quoted token in that message: it contains a space, and the space is not the one you typed. A zero-width space in the same place gave the identical Python message, and in Node's message the token looks empty, because it has no width at all.

If a message says "Expecting value" and the character at the reported position seems to be a space or nothing at all, assume a hidden character.

Seeing what the parser sees

The way to stop guessing is to print the characters that are not plain ASCII, with their names. This works on any text:

import sys, unicodedata

text = open(sys.argv[1], encoding="utf-8").read()
for i, ch in enumerate(text):
    if ord(ch) > 126 or (ord(ch) < 32 and ch not in "\n\r\t"):
        line = text.count("\n", 0, i) + 1
        print(f"line {line:>3} char {i:>5}  U+{ord(ch):04X}  {unicodedata.name(ch, 'UNKNOWN')}")

On the curly-quote example it printed one line for each quote with names that start LEFT DOUBLE QUOTATION MARK and RIGHT DOUBLE QUOTATION MARK, which settles the question in a second. Run it before you try any repair; it shows the full set of foreign characters, not only the first one that the parser tripped on. A file with a curly quote on line 3 usually has another on line 9, and the parser would have shown you each in turn, one fix per run.

A command-line version without a script is grep -nP '[^\x00-\x7F]' file.json, which lists the lines that contain any non-ASCII byte. It cannot tell a legitimate é from a rogue quote, but on a file that should be pure ASCII the output is the answer.

A repair that leaves the content alone

The obvious repair is a replacement of each curly quote with a straight one. Here it is, and here is the catch:

fixed = text.translate({0x201C: '"', 0x201D: '"', 0x2018: "'", 0x2019: "'"})

On {“name”: “Ada”, “age”: 36} it produced a document that parsed to {'name': 'Ada', 'age': 36}. That works because every quote in the file was structural. Now take a valid document with curly quotes inside a string value, {"name": "Ada", "note": "say “hi”"}. Python parsed it as it stood and returned {'name': 'Ada', 'note': 'say “hi”'}. The curly quotes are text, and the author meant them. Run the translate over that file and the result is "say "hi"", which no longer parses, and if it did, the data would be changed.

Therefore the replacement has to be limited to the places where a quote is structural, and a regular expression cannot reliably know where those are. There are three workable approaches:

1. Translate only when the parse fails, and only the characters that matter. For a one-off paste from a document that you know has no curly quotes in its data, translating everything is acceptable. Check the result by reading it.

2. Replace non-breaking and zero-width spaces everywhere. These are never legitimate between tokens, and inside a string a zero-width space is almost never intended. text.replace(" ", " ") and removing U+200B and U+FEFF is safe in the common case. When a value legitimately contains a no-break space, as French typography does before a colon, restrict the replacement to positions outside strings.

3. Use a tolerant parser to load, then write strict JSON. A JSON5 reader does not accept curly quotes either, so this is for other problems; it is mentioned in fixing JSON that is really a JavaScript object literal.

If you want a repair that is correct by construction, walk the text once with a small state machine that tracks whether you are inside a string. Outside a string, convert curly double quotes to straight ones and non-breaking spaces to plain spaces. Inside a string, leave everything. That is about twenty lines and it is the version to put in a script that will be run on files you have not read.

Why the characters appear

It helps to know which channel did it, because you can often turn it off.

• Word processors and note apps autocorrect quotes as you type. Pasting from them carries the converted characters. Paste into a plain-text editor first, or use "paste as plain text", which keeps the curly characters but strips formatting; the quotes are converted on typing, so the real fix is to write code in a code editor.

• Chat applications convert quotes in ordinary text but not inside code blocks (fenced with backticks). Send JSON as a code block, and ask others to do the same.

• Documentation pages show code in pre blocks; copying from the rendered page normally yields the plain characters. If copying fails, view the page source or use the raw version on the code host.

• Email clients rewrap long lines and sometimes convert quotes. Attach the file instead of pasting it.

• PDF and slide decks replace characters with font glyphs on export, and a copy from a PDF may bring ligatures, line breaks in the middle of strings and hyphens. Treat text from a PDF as retyped.

When you control the channel, change it. When you do not, the character names script is your safety net.

A habit that saves time

When a pasted document fails and the structure looks right, do these in order: run the non-ASCII listing, look for names starting LEFT or RIGHT, NO-BREAK, ZERO WIDTH; then re-run the parse and find the next error. Do not edit by eye. Retyping the quotes in an editor with a proportional font can reintroduce the problem if the editor autocorrects. The JSON formatter on this site highlights the failing position, which is a quick second opinion that does not leave the browser.

If you maintain documentation that includes JSON, put a parse step in the build for every code block labeled json. The check takes seconds and prevents the page from shipping an example that nobody can paste. The same discipline is behind finding the exact line of a JSON syntax error and the wider guide to JSON parse errors.

When translating characters is the wrong fix

Translating curly quotes into straight ones is a repair for a document that is already damaged. It is the wrong approach in three situations.

When the data contains typographic quotes on purpose. Product names, article titles, user comments and translated text legitimately hold curly quotes and apostrophes. A blanket translation over such data changes it, and the change is hard to notice until a name like O’Brien is stored as O'Brien in one system and not in another, and a join between them silently loses rows. Compare against the original when the data is anything but a configuration.

When the source is a database or an API. If the text comes from a system you can query, fix it there. A column with U+00A0 in it is a data quality problem, and cleaning the symptom in every consumer is how five copies of the same fix come to exist.

When the text came from untrusted users. Silent normalization of input can change its meaning, and for identifiers (usernames, email addresses, domain names) the difference between a lookalike character and an ASCII one is a security issue. Reject the input with a precise message instead.

For the Unicode categories behind these characters see the Unicode Standard's general category list, and for the whitespace JSON allows, section 2 of RFC 8259.