What JSON Allows and What Parsers Quietly Accept in Practice

JSON is the one data format that every developer thinks they understand. It has six kinds of value, a grammar that fits on the back of a business card and a name that suggests it is just JavaScript. Then a config file with a trailing comma works in one tool and fails in another, a service returns an ID that is off by three in the last digit, and two systems read the same document and disagree about what it says. The format is simple. The edge cases, the places where the specification says "may" or says nothing, are where parsers go their own ways, and most real JSON bugs live there. This guide lays out what the grammar allows, then shows what five common parsers do with the inputs at its edges, based on a 26-case experiment, and ends with how to write and read JSON so that those differences do not matter. The tooling around the format, including the command-line side, is surveyed in JSON parse errors and how to read them, and the detailed cheat sheet of test inputs is in a cheat sheet of JSON edge cases to throw at a parser.

What the grammar allows

JSON is defined by RFC 8259 and, in nearly identical terms, by ECMA-404. A value is one of six things: an object, an array, a string, a number, true or false, or null. That is the whole type system; there is no date, no binary, no integer as distinct from a float, no comment and no way to say "undefined".

An object is braces around zero or more "name": value pairs separated by commas, and the name must be a string in double quotes. An array is brackets around zero or more values separated by commas. A string is double quotes around characters, with the escapes \", \\, \/, \b, \f, \n, \r, \t and \uXXXX, and with raw control characters (below U+0020) forbidden: a literal tab or newline inside a string is not allowed. A number is an optional minus, an integer part with no leading zeros (0 alone, or a digit 1 to 9 followed by digits), an optional fraction that needs digits on both sides of the point, and an optional exponent. Whitespace, meaning space, tab, newline and carriage return, may appear between tokens.

Since RFC 7159, a document can be any value, so 42 and "hello" are valid JSON texts on their own, not only objects and arrays. The older RFC 4627 allowed only the two containers at the top. Nothing in the grammar allows trailing commas, comments, single-quoted strings, unquoted names, NaN, Infinity, hexadecimal, a leading plus sign, a number like .5 or 1. or a second document after the first. A parser that accepts any of those is accepting more than the format, and that is where the tour begins.

The RFC is also explicit that parsers have latitude. It says that an implementation may set limits on the size of texts, the depth of nesting, the range and precision of numbers and the length of strings, and that a parser may accept non-JSON forms. So "not valid JSON" is not a property of the bytes alone: it is a statement about a parser, and the next section shows how much parsers vary.

The experiment

I wrote 26 inputs, each a small file with exact bytes, and asked five parsers whether they accept it. The parsers were Python's json.loads on text, Node's JSON.parse, PHP's json_decode, jq 1.6 and DuckDB's json_valid. Here are the results for the inputs that matter most, with ok for accepted and ERR for rejected:

The agreement is on the clear cases: single quotes, unquoted names, comments, raw control characters, hex numbers and bad escapes were rejected everywhere, and top-level scalars, huge exponents and big integers were accepted everywhere. Everything else splits the field.

Some details need caveats. For Python, the BOM case depends on the input type: json.loads on a string beginning with the BOM character raises an error ("Unexpected UTF-8 BOM"), and on bytes it detects the BOM and accepts the document, which is why the same file can parse in a script that reads bytes and fail in one that reads text. jq accepted the two-documents input because it is a stream processor that reads values one after another, and accepted empty input because it printed nothing. DuckDB's json_valid is strict about numbers and quotes and lenient about commas, NaN and Infinity. PHP rejected the deeply nested input because json_decode defaults to a maximum depth of 512, which is a parameter and not a failure of the grammar.

What the values come out as

Acceptance is half of it. The values that come out differ too. For the same four inputs, Python, Node, PHP and jq produced:

Duplicate keys resolve to the last value in all four, a convention and not a rule: the RFC says names within an object "SHOULD be unique" and that behavior with duplicates is unpredictable. DuckDB's JSON type keeps the original text, duplicates included, until you extract a value. The overflowing exponent becomes infinity in Python, null after Node serializes it back and the largest double in jq. The 23-digit integer is exact in Python, which has arbitrary-precision integers, and a rounded float everywhere else.

The last row is the one that bites. 9007199254740993 is 2 to the 53rd plus one, the first integer that a 64-bit float cannot represent. Node and jq read it as the float and print 9007199254740992, off by one, and the reader has no way to know. PHP kept the exact 64-bit integer, and Python kept it as an arbitrary-precision one. If that number is an order ID, the same document identifies two different orders in two services (how JSON handles big numbers and why IDs lose digits has the fixes).

Why the differences exist

JSON was specified after it was in use. Douglas Crockford described it as the subset of JavaScript literals that was safe to exchange, and implementations grew from eval and from hand-written parsers that accepted what their authors thought was reasonable. Standards then wrote down what the strictest reading allowed, and left a lot to implementations. Every deviation in the table has a reason: Python allows NaN and Infinity because they are Python floats and json.dumps writes them by default; jq 1.6 was lenient about numbers because its parser accepts anything that looks like a numeric literal; DuckDB reads trailing commas because its JSON reader, built for analytics on messy files, is deliberately forgiving.

The consequence is that a document can be valid for the system that wrote it and invalid for the one that reads it. A Python service that emits NaN for a missing measurement produces output that a JavaScript client cannot parse (is NaN or Infinity valid in JSON). A config file with a trailing comma that a lenient tool accepts breaks the first strict tool in the pipeline (trailing commas in JSON and why parsers reject them).

Reading the table as a decision aid

The table is useful as a way to choose what to write. Take the rows where all five parsers agree on "reject": single quotes, unquoted names, comments, raw control characters, hexadecimal. Writing any of those is a guaranteed failure everywhere, and the fix is to stop. Take the rows where they split on rejection, trailing commas, NaN, leading zeros, a BOM and lone surrogates. Those are the risky ones: they work in your tests, because your parser tolerates them, and fail in production, because the consumer is stricter. The safe response is to treat the strictest parser's reading as the standard. Take the rows where all five accept but the values differ, duplicate keys and big integers: nothing fails and the data is quietly different. Those are the most dangerous, since no error ever tells you.

That three-way sort, fails everywhere, fails somewhere, works everywhere with different meaning, is how I would triage any new JSON producer or consumer. Run the cases, put each in a bucket and fix the second and third first, because the first announces itself.

Strings, escapes and encodings

The RFC says that JSON exchanged between systems that are not part of a closed ecosystem must be encoded in UTF-8, and that implementations must not add a byte order mark, though they may ignore one when reading. That is the specification, and it explains the BOM row in the table: Python on bytes ignores it, and on text raises, Node and PHP reject it, jq and others skip it. A producer that writes a BOM is violating a MUST NOT, and a consumer that rejects it is within its rights, so the practical answer is to never write one and to strip one defensively when you read files that came from Windows editors.

Escapes make the second source of trouble. \u00e9 and the raw character é are the same string. A character outside the Basic Multilingual Plane, such as the grinning face U+1F600, is written in an escaped form as a surrogate pair, \ud83d\ude00, two escapes standing for one character. Python's json.dumps writes that form by default (ensure_ascii=True) and the raw character with ensure_ascii=False; both parse back to the same string. The problem is a surrogate escape on its own, \ud800 with no partner. The grammar allows it, since it is a valid escape sequence, and it names no character. Python and Node accept it, PHP and jq reject it, and a library that then tries to encode the string as UTF-8 will fail or write a replacement character. Reject lone surrogates at the boundary (unicode escapes and surrogate pairs in JSON).

String length shows the same split in another form. The grinning face is one character, four bytes in UTF-8 and two units in UTF-16. Python's len returned 1, jq's length returned 1, and JavaScript's .length returned 2, because JavaScript counts UTF-16 code units. A validation rule of "at most 20 characters" therefore means different things in a Python service and in a JavaScript form, and an emoji-heavy name fits in one and not the other. Count what the rule means and say which unit in the specification.

Numbers have no range in the grammar

The number grammar says nothing about size or precision. 1, 1.0 and 1e0 are all legal, and a parser decides what to do with them. Python keeps the distinction: json.loads("1.0") is a float and json.loads("1") is an integer, and json.dumps(-0.0) writes -0.0. JavaScript has only one number type, so JSON.parse("1.0") and JSON.parse("1") are both 1, and JSON.stringify(-0) writes 0. A round trip through JavaScript therefore turns 1.0 into 1, which is harmless unless a downstream system reads 1.0 as a decimal with a scale. The size problem was shown above: any integer above 2 to the 53rd is at risk in a parser that uses doubles.

Treat numbers that must be exact, money, identifiers, counters beyond 9 quadrillion, as strings, or as integers in a documented range, and test the path through every parser that reads them.

A strict reader in a few lines

You can make Python's parser as strict as the specification with two hooks, one for the constants and one for object pairs, and a defensive BOM strip:

import json

def reject_constant(name):
    raise ValueError(f"non-standard number {name} is not allowed")

def reject_duplicates(pairs):
    seen = {}
    for key, value in pairs:
        if key in seen:
            raise ValueError(f"duplicate key {key!r}")
        seen[key] = value
    return seen

def strict_loads(text):
    if text.startswith("\ufeff"):
        text = text[1:]
    return json.loads(text, parse_constant=reject_constant, object_pairs_hook=reject_duplicates)

Run on five inputs:

'{"a": 1, "b": [true, null]}'        -> {'a': 1, 'b': [True, None]}
'[NaN]'                              -> rejected: non-standard number NaN is not allowed
'{"a": 1, "a": 2}'                   -> rejected: duplicate key 'a'
'{"a": 1,}'                          -> rejected: Illegal trailing comma before end of object: line 1 column 8 (char 7)
'\ufeff{"ok": true}'                 -> {'ok': True}

The valid document passes, NaN and the duplicate are rejected by the hooks, the trailing comma was already rejected by the parser (Python 3.13 words the error as "Illegal trailing comma before end of object"), and the BOM-prefixed document passes because the function strips it. About fifteen lines turn a lenient default into the strict contract. The equivalent hooks exist in other languages: a reviver or a custom parser in JavaScript, JSON_THROW_ON_ERROR and a depth argument in PHP, and a schema validator in front of everything.

Whitespace, key order and what "the same document" means

Two JSON texts can be different byte for byte and the same document. Whitespace between tokens is free, and the order of keys in an object has no meaning in the specification. In Python, json.dumps({"b": 1, "a": [1, 2]}) gives {"b": 1, "a": [1, 2]}, and the compact form of the same data with sorted keys is {"a":[1,2],"b":1}: the two parse to equal values and compare unequal as text. That matters whenever you hash, sign or diff JSON. A hash of the text changes if a library changes its spacing or key order, so a signature needs a canonical form. RFC 8785, the JSON Canonicalization Scheme, defines one (sorted keys, no extra whitespace, a specified number format), and some libraries implement it. For diffs, sort the keys and indent consistently before comparing, or use a tool that compares parsed values.

A checklist for an API that sends and receives JSON

For what you send: UTF-8, no BOM, unique keys, finite numbers only, IDs and exact decimals as strings, timestamps as ISO 8601 strings with an offset (JSON dates and the lack of a date type), a declared content type of application/json and nothing before or after the document. For what you receive: a size limit before parsing, a depth limit, a strict parser configuration as above, schema validation after parsing, an explicit decision about unknown fields (ignore or reject) and the same error shape for every kind of failure, so that clients can tell malformed JSON from valid JSON that violates the rules. Document both, with an example payload that a client can paste, and keep a conformance test with the awkward inputs next to the code that parses them.

How the leniency crept in

It helps to know why the format is both tiny and untidy. JSON started as a way to send data to a browser: the objects and arrays were already valid JavaScript, so a page could read a response with eval. Douglas Crockford named and described the subset, and the first spec, RFC 4627 in 2006, was a few pages. Early parsers were small and written by people who wanted them to work on the data they had, so they accepted a little more than the grammar. When languages other than JavaScript adopted the format, their authors made their own choices about what to do with NaN, with integers beyond the float range and with repeated names, and each choice became behavior that existing programs depended on. Later standards could tighten the text and could not change a million deployed parsers.

The effect is that strictness is a property of a particular parser at a particular version, and almost never an accident of the format. A parser that is lenient today has users who rely on it, and a parser that is strict is easier to reason about and breaks files that used to work. Neither is wrong, and neither is safe to assume.

Large documents and streams

One more edge is size. The grammar allows a document of any length, and most parsers build the whole structure in memory, so a 2 GB array of records needs several gigabytes of RAM to read, and a parse that fails at the last byte has wasted all of it. For large collections, the common alternative is one JSON value per line, called JSON Lines or newline-delimited JSON, where each line is a complete document. A reader handles a line at a time, a writer can append, and a damaged line affects one record and not the file. DuckDB reads it with read_json and read_ndjson, jq processes it as a stream, and Python's loop over lines with json.loads does the same in a few lines. When you control the format of a large export, choose JSON Lines over one big array (converting CSV to JSON Lines for streaming shows the conversion). When you receive a big array, use a streaming parser, such as ijson in Python or jq --stream, instead of loading it.

JSON beside its neighbors

Choosing JSON for data exchange is a choice among formats, and each has its own edges. CSV is flat, typeless and universal in spreadsheets, and has no standard for nesting. XML has schemas, namespaces, comments and attributes, and a verbose syntax with its own parser pitfalls. YAML is a superset in spirit that allows comments, anchors and many ways to write a value, with the implicit typing that turns no into a boolean and 1.10 into a number, which is why it is a poor format for data from outside. TOML and INI suit small configuration files. JSON sits in the middle: typed enough for APIs, small enough to parse everywhere, with no comments and no date. If you need to compare, look at what each does to your awkward values: a long ID, a date, a missing value, a text with a quote and a newline. The format that preserves them with the least ceremony is the right one for that data.

What to do with a document that is almost JSON

Most of the trouble people report is not a spec question. It is a file that is nearly JSON: a trailing comma left by a hand edit, single quotes from a Python repr, a JavaScript object literal pasted into a file, a truncated response, an HTML error page where JSON was expected, a BOM from an editor. Each has a specific diagnosis and a specific fix, and the error message points at it when you know how to read it. The next articles in this series go through them: reading parse error messages, single quotes, trailing commas, comments and the others. Repair is a last resort: where you can, fix the producer so that the documents are valid, and where you cannot, use a purpose-built lenient parser such as JSON5 for files that you know come from humans, rather than regular expressions that guess.

When differences become security problems

Disagreement between parsers is not only an annoyance. If two components read the same JSON and take different values from it, an attacker can write a document that one component validates and the other acts on. Duplicate keys are the classic case: {"role": "user", "role": "admin"} validated by a parser that keeps the first value and executed by one that keeps the last. A lone surrogate, a huge number or a differently handled BOM can do similar work. Nicolas Seriot's survey, Parsing JSON is a Minefield, tested dozens of parsers against hundreds of inputs and documented how much they vary, which is the larger version of the experiment here.

Defenses are practical. Parse once, at the boundary, and pass the parsed structure inward, never the text. Reject duplicate keys where you can (in Python through object_pairs_hook, shown in duplicate keys in a JSON object and which one wins). Validate against a schema after parsing. And limit size and depth, since a deeply nested document can exhaust a parser's stack.

Writing and reading safely

The advice splits by direction. When you produce JSON, produce the strict, boring subset that every parser agrees on. RFC 7493, the I-JSON profile, is a written version of it: object member names unique, numbers within the range of an IEEE double, no non-finite numbers, UTF-8 and no unpaired surrogates, and dates and binary data as strings in agreed encodings. Write IDs above 2 to the 53rd as strings, write times as ISO 8601 strings, omit fields instead of writing NaN, and write documents with a library and never by hand-concatenating strings.

When you consume JSON, be as strict as your needs allow, and say so. Use the parser's strict mode and turn off non-standard extensions (allow_nan=False in Python's json.loads with a parse_constant that raises). Handle the BOM on purpose, since files saved by Windows tools often have one. Decide what to do about duplicates. Limit depth and size. And read errors properly: the message tells you the position and the expectation, and fixing JSON parse errors by reading the message properly shows how to use them.

Test your own parser

If JSON from outside enters your system, test your parser with the awkward inputs, because its behavior is the one that matters. Keep the 26 inputs, or a larger set such as the JSONTestSuite, in a folder, run them through the code path you use in production and write down the accept and reject list. Check it again when you upgrade the library or the runtime. A cheat sheet of JSON edge cases to throw at a parser has the set, the scripts that produced the table above and the expected results for each case. A parser's behavior at these edges is stable for years and changes without much notice when a major version arrives, so a test that fails on upgrade is useful information.

When strictness is the wrong goal

Strictness is for data exchanged between programs. For files that people edit, configuration, settings, fixtures, a format with comments and trailing commas is kinder, and the right move is to choose one openly instead of bending JSON: JSONC for editor settings, JSON5 for human-edited config, YAML or TOML where those fit (JSON versus JSON5 versus JSONC and when to use each compares them). A file with comments in a .json file works in the tool that expects it and is a bug waiting in any other. The authority for the grammar is RFC 8259, with the interoperability profile in RFC 7493, and Python's behavior is documented in the json module reference.