Reading JSON From an API With requests and Handling Errors

Reading JSON from an API looks like one line, requests.get(url).json(), and it works in the demo. In production that line is where an empty body, a gateway error page, a rate limit and a login redirect all end up as the same exception. This article runs the failure cases against a small local server I wrote for the purpose, shows exactly what requests raised for each, and builds the pattern I would put in front of any JSON API: status check, timeout, retries, then parse, with the error body kept for the log.

One exception, five causes

I requested five paths and called .json() on each response. The first returned {"id": 1, "name": "a"} and parsed. The rest:

200 with an empty body            JSONDecodeError: Expecting value: line 1 column 1 (char 0)
204 No Content                    JSONDecodeError: Expecting value: line 1 column 1 (char 0)
502 with an HTML error page       JSONDecodeError: Expecting value: line 1 column 1 (char 0)
200 with an HTML login page       JSONDecodeError: Expecting value: line 1 column 1 (char 0)
200 with a UTF-8 BOM prefix       JSONDecodeError: Unexpected UTF-8 BOM (decode using utf-8-sig): line 1 column 1 (char 0)

Four different situations, one identical message. The parser saw a first character that is not JSON, and that is all it can say. The empty 200 is a bug on the server, the 204 is correct behavior (the API told you there is nothing to parse), the 502 is an outage and the login page is an authentication problem that your client followed through a redirect. If you only log the exception, you cannot tell which one woke you up. That is the main thing to fix.

The same message appears when you parse text yourself, and Python's JSONDecodeError: Expecting value at line 1 column 1 goes through the underlying causes. Here the point is that a network client has more information than the parser does, and you should use it before parsing.

Check the status before the body

response.raise_for_status() raises requests.HTTPError for any 4xx or 5xx. In the tests it turned the 502 into HTTPError: 502 Server Error: Bad Gateway for url: http://127.0.0.1:8765/html502 and a 404 into 404 Client Error: Not Found for url: ..., and it does nothing for a 200 or a 204. The exception carries the response, so the API's own error body is still available:

try:
    r = requests.get(url, timeout=5)
    r.raise_for_status()
except requests.HTTPError as e:
    print(e.response.status_code, e.response.json())
# 404 {'error': 'not found', 'code': 'E404'}

Note that the last line calls .json() inside the handler, on a response the server gave a JSON content type. For the HTML 502 that same call would raise again. Guard it: check e.response.headers.get("Content-Type", "").startswith("application/json") before parsing an error body, and otherwise log the first few hundred characters of e.response.text. An error handler that raises its own error is the worst kind of failure to read in a log.

For the 204, check the status or the body length before calling .json():

if r.status_code == 204 or not r.content:
    return None
return r.json()

An API that returns an empty body with a 200 is breaking its own contract, and returning None there hides it. Whether you want that depends on the API, but make it a decision in the code.

For the login page case, look at the content type. A 200 with text/html where you expected application/json is almost always an auth redirect or a proxy page. The check costs one line and turns a mystery into a message you can read: Expected JSON, got text/html from <url>: <first 200 chars>.

Which exception to catch

The exception requests raises from .json() is requests.exceptions.JSONDecodeError. Its base classes, as printed from the class hierarchy in 2.34.2, are:

requests.exceptions.JSONDecodeError
  requests.exceptions.InvalidJSONError
  requests.exceptions.RequestException
  builtins.OSError
  json.decoder.JSONDecodeError

It inherits from RequestException (so a broad except requests.RequestException catches it) and from the standard library's json.JSONDecodeError, so except ValueError and except json.JSONDecodeError catch it too, and e.pos, e.lineno and e.colno exist. That inheritance was added so that code written against the standard library keeps working whichever JSON library requests happens to use. A typical handler catches requests.RequestException last, after the specific cases, and reports each differently.

Timeouts: the default is forever

requests has no default timeout. A call without one can hang indefinitely on a server that accepts the connection and never answers. I pointed timeout=0.5 at an endpoint that slept for two seconds and got requests.exceptions.ReadTimeout: ... Read timed out. (read timeout=0.5). A tuple sets connect and read separately, timeout=(3.05, 10). Two things to know:

• The read timeout is the time between bytes, not the total time. A server that dribbles a byte every few seconds never trips it. If you need a hard deadline, wrap the call in your own deadline logic or stream with a total budget.

• A refused connection is a different exception. Connecting to a closed port raised requests.exceptions.ConnectionError ... Max retries exceeded with url: /x, with the cause nested inside the message. Catch ConnectionError and Timeout separately from HTTPError, since the first two mean you never got a response at all.

Set the timeout on every call. A wrapper function or a Session subclass that injects it makes forgetting impossible.

Retries with a Session

A Session reuses connections, and it is where retries belong. The tests used an endpoint that returned 503 twice, then 200:

from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

s = requests.Session()
s.mount("https://", HTTPAdapter(max_retries=Retry(
    total=4, backoff_factor=0.5, status_forcelist=[429, 502, 503, 504])))
r = s.get(url, timeout=5)

The server counter showed 3 requests and s.get(...).json() returned {'ok': True}: two retries, then success, with no code in the caller. status_forcelist lists the statuses to retry, backoff_factor sets the growing delay between attempts, and by default retries apply only to idempotent methods, so a POST is not repeated unless you opt in with allowed_methods. That default is the right one. Retrying a payment POST because of a gateway timeout can charge twice.

When all status retries fail, you get a requests.exceptions.RetryError, not an HTTPError. And the rate limit case deserves attention: a 429 came back with Retry-After: 1 in the headers. Retry honors that header by default (for 413, 429 and 503), so listing 429 in status_forcelist makes the client wait as the server asked: a second test with a 429 followed by a 200 took 1.0 s for two requests with backoff_factor=0, which is the server's one second and none of ours. An endpoint that kept answering 503 exhausted total=2 after 3 requests and 2.0 s, raising RetryError. If you handle it yourself, read r.headers["Retry-After"], which is seconds or an HTTP date.

Encoding, BOMs and odd numbers

requests decodes .json() using the response's detected encoding. For the BOM case, r.encoding was utf-8 while apparent_encoding reported UTF-8-SIG, and .json() raised Unexpected UTF-8 BOM (decode using utf-8-sig). The fix is json.loads(r.content.decode("utf-8-sig")), or better, ask the producer to stop writing it (why JSON parse fails on a byte order mark). A JSON body with non-ASCII text ({"n": "café"}) parsed correctly.

The standard parser also accepts what JSON does not allow: a body of {"v": NaN} returned {'v': nan}. If a number must be a real number, validate after parsing, or pass parse_constant by calling json.loads(r.text, parse_constant=...) yourself. On the way out, json= with float("nan") raised InvalidJSONError: Out of range float values are not JSON compliant: nan, which is the check you want, since it stops an invalid payload before it leaves.

Sending with json={"a": 1} serialized {"a": 1} and set Content-Type: application/json, so do not also pass headers or data=json.dumps(...) for the same request.

Pagination and large responses

A loop that follows a next link read three items across two pages in the test ([1, 2, 3], stopping when next was null). Add a page cap and a timeout per request, and check the status on each page. For responses too large to hold, use stream=True and a streaming parser (handling large JSON files in Python with ijson); r.json() reads the entire body into memory first.

A complete pattern

def get_json(session, url, **kw):
    r = session.get(url, timeout=kw.pop("timeout", 10), **kw)
    r.raise_for_status()
    if r.status_code == 204 or not r.content:
        return None
    ctype = r.headers.get("Content-Type", "")
    if "json" not in ctype:
        raise ValueError(f"expected JSON from {url}, got {ctype!r}: {r.text[:200]!r}")
    return r.json()

Each line answers one failure from above: status, empty body, wrong content type, then parse. Once parsed, validate the shape before trusting it (validating JSON with jsonschema in Python), or build typed objects from it (parsing JSON into typed objects with pydantic). To look at a payload while debugging, paste it into the JSON formatter.

When requests is the wrong client for this job

You need async concurrency. requests blocks. For hundreds of parallel calls use httpx or aiohttp; the status, timeout and retry thinking is the same.

You need HTTP/2 or fine-grained connection control. requests speaks HTTP/1.1.

The response is a stream. For server-sent events or NDJSON feeds, iterate lines with stream=True and parse each one (reading and writing JSON files in Python covers the line-by-line pattern).

The API ships an SDK. A maintained client already handles auth, pagination and retries for that service.

You are testing. Mock the HTTP layer (responses, requests-mock) so tests do not depend on the network.

The documented behavior is in the requests quickstart and the advanced usage guide, and the retry options are in the urllib3 Retry reference.