Fixing JSON With Unescaped Newlines in Strings Safely

A text field with a line break goes into a hand-built JSON string, and the next consumer cannot read it. The error names a "control character" or a "bad escape", and the document looks fine on screen because the editor shows the line break as a line break. This article explains which characters JSON strings may not contain, what Python and Node say when one appears, how to fix the data safely, and why the permissive switch that makes the error go away is a trap you should use with eyes open.

What a string may hold

The JSON grammar says a string is a double quote, then any number of characters, then a double quote, and the characters may be anything except the double quote, the backslash and the control characters U+0000 through U+001F. Those thirty-two characters include the newline (U+000A), the carriage return (U+000D) and the tab (U+0009). To put one in a string you write an escape: \n, \r, \t, or the general \u000a form. The backslash and the quote are escaped as \\ and \".

So this is not valid JSON, because the string contains an actual line feed in the middle:

{"note": "line one
line two"}

and this is the same data written correctly:

{"note": "line one\nline two"}

The rule exists so that a parser can read a string in one pass without ambiguity, and so that line-oriented tools can rely on a newline never being inside a value (the property that JSON Lines depends on, discussed in what JSON Lines is and why pipelines prefer it).

What the parsers print

For {"note": "line one<LF>line two"} with a real line feed, Python 3.13 raised:

JSONDecodeError: Invalid control character at: line 1 column 19 (char 18)

Column 19 is the line feed itself, so the position is useful: it points at the first bad character. A literal tab in a string gave the same message at its own position. Node 24 said:

SyntaxError: Bad control character in string literal in JSON at position 8 (line 1 column 9)

The words differ and the meaning is the same. In both messages the position points inside the string, at the offending character, which is more precise than most syntax errors. If you see "control character", find the position, print repr(text[pos - 20 : pos + 20]) and you will see \n or \t in a place it should not be.

A related error is a backslash that begins an escape that does not exist. A Windows path pasted into JSON as "C:\Users\ada" fails with Invalid \escape: line 1 column 10 (char 9) in Python and Bad escaped character in JSON at position 9 in Node, because \U is not an escape (only \", \\, \/, \b, \f, \n, \r, \t and \uXXXX are). Written correctly, the path is "C:\\Users\\ada", and it parsed to C:\Users\ada. A backslash followed by a valid letter, as in "C:\temp", is worse: it parses without error, and \t turns into a tab, which makes the path wrong in a way nobody notices until the file is not found.

Where unescaped newlines come from

Almost always the JSON was built by gluing strings together. A template like '{"note": "' + note + '"}' works until note contains a quote, a backslash or a line break. The usual places:

• Shell scripts using echo "{\"note\": \"$NOTE\"}" with a value that holds a newline.

• Templating in configuration management or a CI pipeline that substitutes a multi-line variable into a JSON file, such as a certificate, a private key, a commit message or a changelog.

• Spreadsheet or database exports where a cell has a line break, and the exporter wrote the cell's text unchanged.

• Hand editing: someone pressed Enter in the middle of a long string to keep a line short, which is natural in a code editor and invalid in JSON.

• Copying from a log that wraps long lines.

A multi-line value belongs in the document as \n characters. PEM keys are the typical example: a key pasted into a JSON field has to become one line with \n between its lines, and a tool that adds the newlines for you is json.dumps or jq --arg.

The fix at the producer

Use a serializer. In Python, json.dumps({"note": "line one\nline two\ttab"}) printed {"note": "line one\nline two\ttab"}, with the newline and tab as two-character escapes, and json.loads of that returned the original string. In JavaScript, JSON.stringify does the same. In a shell script, let jq build the document:

jq -n --arg note "$NOTE" '{note: $note}'

--arg passes the value as a string and jq escapes whatever is in it. The shell never builds JSON by concatenation, so quotes, backslashes and line breaks in $NOTE are all safe. This is the approach covered further in passing shell variables into jq safely.

If a template engine fills in the field, give it a JSON-encoding filter (tojson in Jinja, to_json in Ansible, json_encode in PHP) and apply it to the value, not to the whole document by eye.

Repairing a file you already have

If you have to read a file you cannot regenerate, the options are these.

**strict=False in Python.** json.loads(text, strict=False) allows control characters inside strings. On my newline example it returned {'note': 'line one\nline two'}, and for the tab, {'note': 'a\tb'}. It is a legitimate option when you are loading data from a system you cannot change and the content really is what it appears to be. It only relaxes control characters and does nothing for a bad backslash, a missing comma or a single quote.

Escape while scanning. A small state machine can walk the text, track whether it is inside a string (watching for unescaped quotes and backslash pairs) and replace raw line feeds and tabs with \n and \t only while inside a string. It is about fifteen lines, and unlike a regular expression it will not damage the structure outside strings. Run the result through the parser to confirm, then write the file back with json.dump so the escaping is correct from then on.

JavaScript has no switch. JSON.parse has no lenient mode. Repair the text first, or use a tolerant parser such as JSON5 or a library that is documented to accept raw newlines, and be aware that "tolerant" means each library chooses its own extensions.

Whatever you use, compare the number of records before and after, and look at a few long values by eye. A repair that turns two fields into one because a quote was missing leaves a document that parses and is wrong.

What the line break means to the data

Escaping solves the syntax. It does not tell you whether the newline should be there. Windows text often carries \r\n, and a field exported from a form may have a trailing newline that nobody wanted. After you fix the encoding, decide what the value is supposed to be: normalize line endings to \n, trim the ends, collapse runs of blank lines. Those decisions are about data, and a JSON parser will faithfully preserve whichever you choose, including the carriage return you did not know was there. The tests I ran showed \n in the parsed string, which is the single line feed; on text from a Windows application you may see \r\n, and comparing strings from two systems that differ only in that will fail in confusing ways.

If your document will be read by a person as well as a program, remember that long strings with \n escapes are hard to read. Some teams store multi-line text in a separate file, referenced by name, and keep the JSON for structure. That is a design choice for configuration, and it avoids the problem altogether.

You can paste a document into the JSON formatter to see the position of the first bad character; it runs in the browser, so the text does not leave your machine. The way to read a position is in finding the exact line of a JSON syntax error.

When strict=False hides a real bug

Allowing control characters on read is convenient and has costs. These are the cases where I would not use it.

When the unescaped newline is a symptom of a broken producer. A string that contains a raw newline probably also contains a raw quote or backslash somewhere, and strict=False will not catch those. You will fix one error to meet the next, and the real defect, string concatenation, stays in place and keeps producing new variants.

When the data crosses a trust boundary. A parser that accepts what the specification forbids behaves differently from every other parser in the chain. If a gateway validates with a strict parser and your service reads with strict=False, a payload can pass one and mean something else to the other. That is the kind of mismatch described in what JSON allows and what parsers quietly accept. Keep the same strictness at each stage.

When the file is going to be shared. The next consumer will not use your switch. A file that only parses with strict=False is a defective file, and it should be corrected and re-published.

When the data is not JSON. A file with one value per line and raw tabs may be a log, a TSV or a mislabeled format. Read it as what it is.

The specification text on strings is in RFC 8259, section 7, and the strict parameter is described in the Python json module documentation.