Converting CSV to JSON With Python Without Losing Types

CSV to JSON is the conversion people write first and revisit later, because the first version works on the sample and the second one is needed when somebody notices that a zip code lost its zero. The two formats disagree about types: CSV has none, and JSON has strings, numbers, booleans and null. Every conversion is a set of decisions about which text becomes which type. This article does the plain conversion, then makes the decisions explicit, then measures six converters on 193,708 rows and reads their output back against the source to see what each one changed.

The plain conversion

The shortest correct conversion reads rows as dictionaries and dumps each one:

import csv, json, sys
src, out = sys.argv[1], sys.argv[2]
with open(src, newline="", encoding="utf-8") as f, open(out, "w", encoding="utf-8") as o:
    o.write("[")
    first = True
    for row in csv.DictReader(f):
        if not first:
            o.write(",\n")
        o.write(json.dumps(row, ensure_ascii=False))
        first = False
    o.write("]\n")

It writes the array one object at a time, so memory does not grow with the file. csv.DictReader uses the header for the keys and yields every value as a string, which is exactly what the CSV contained. ensure_ascii=False keeps Zoë as Zoë instead of Zoë, which is valid either way and half the size for text in many languages. The first and fifth rows of the small file:

python lab/conv/t0145.py

1. DictReader straight to JSON (every value is a string)
{"id": "1", "name": "Ana Silva", "zip": "02134", "score": "12.50", "active": "true", "joined": "2025-03-01", "note": "plain"}
{"id": "5", "name": "Di", "zip": "01002", "score": "3.14", "active": "false", "joined": "2025-03-05", "note": ""}

2. A typed conversion (id int, zip text, score number, active boolean, empty is null)
{"id": 1, "name": "Ana Silva", "zip": "02134", "score": 12.5, "active": true, "joined": "2025-03-01", "note": "plain"}
{"id": 4, "name": "Cy", "zip": "90210", "score": null, "active": true, "joined": "2025-03-04", "note": "line one\nline two"}
{"id": 5, "name": "Di", "zip": "01002", "score": 3.14, "active": false, "joined": "2025-03-05", "note": null}
{"id": 6, "name": "Ed", "zip": "60601", "score": 100.0, "active": null, "joined": "2025-03-06", "note": "NO"}
{"id": 8, "name": "Fay", "zip": "1e5", "score": -5.25, "active": true, "joined": "2025-03-08", "note": " padded "}

3. What float() does to a decimal on the way out
{"score": 1234.5, "score2": 1.1, "big": 0.3}
as a string instead: {"score": "1234.50"}

4. pandas.DataFrame.to_json
{'id': dtype('int64'), 'name': <StringDtype(na_value=nan)>, 'zip': dtype('float64'), 'score': dtype('float64'), 'active': <StringDtype(na_value=nan)>, 'joined': <StringDtype(na_value=nan)>, 'note': <StringDtype(na_value=nan)>}
[{"id":1,"name":"Ana Silva","zip":2134.0,"score":12.5,"active":"true","joined":"2025-03-01","note":"plain"},{"id":2,"name":"O'Neil, Pat","zip":10001.0,"score":0.0,"active":"false","joined":"2025-03-02","note":"has, a comma"}]
[{"id":"1","name":"Ana Silva","zip":"02134","score":"12.50","active":"true","joined":"2025-03-01","note":"plain"},{"id":"2","name":"O'Neil, Pat","zip":"10001","score":"0","active":"false","joined":"2025-03-02","note":"has, a comma"}]
[{"a":0.3,"p":"a\/b"},{"a":0.3333333333,"p":"c"}]

The first block shows the result: "id": "1", "score": "12.50", "active": "true", and "note": "" for the empty note in row 5. Nothing was lost, and nothing was interpreted. Every number is a string, which most consumers of JSON will not enjoy. This is still the right output when you cannot yet say what the columns mean: it is a faithful copy that you can type later.

Typing is a decision per column

The second block applies a small conversion function:

• id becomes an integer.

• zip stays a string. A zip code, a phone number or an account number is an identifier. You never add two of them, and a number type would drop the leading zero (02134 is not 2134).

• score becomes a JSON number, or null when the CSV field was empty.

• active becomes true, false or null, using a table of spellings (true, yes and false here).

• joined stays an ISO 8601 string. JSON has no date type.

• An empty note becomes null. Whether empty means null or the empty string is a contract question, and the output makes it visible: row 5 has "note": null, and row 6, whose note is NO, keeps the string "NO" (a trap for the YAML article, not for JSON).

The conversion is code you can read, and every line of it is a statement about the data. That is the point: a generic converter makes the same decisions without telling you.

Numbers are doubles

RFC 8259 does not say what a number is, and most parsers read it as a double. The third block of the output shows the effect: 1234.50 became 1234.5, 1.10 became 1.1 and 100 became 100.0 in Python's rendering. A trailing zero carries no information in a double, but it does in a price, a version or an id. If the scale matters, write the number as a string ("1234.50"), which the output also shows. If you must write a number, make it the same one the consumer will read: a double has about 15 to 17 significant digits, so a value over 2^53 (9,007,199,254,740,992) does not survive.

What pandas does

pandas is the usual shortcut, and its to_json makes its own decisions, which are visible in the last block:

print("\n4. pandas.DataFrame.to_json")
df = pd.read_csv(SRC)
print(df.dtypes.to_dict())
print(df.head(2).to_json(orient="records"))
df2 = pd.read_csv(SRC, dtype=str, keep_default_na=False)
print(df2.head(2).to_json(orient="records", force_ascii=False))
print(pd.DataFrame({"a": [0.1 + 0.2, 1 / 3], "p": ["a/b", "c"]}).to_json(orient="records"))

Reading with defaults made zip a float64, because one value in the column is 1e5, which parses as a float. Every other zip in the column became a float with it, and to_json wrote 02134 as 2134.0. No error, no warning. The same call with dtype=str, keep_default_na=False kept "02134". Two more things in the output: to_json rounds floats to 10 significant digits by default (0.1 + 0.2 came out as 0.3 and 1/3 as 0.3333333333, both because of the double_precision option), and it escapes the forward slash, writing a/b as "a\/b". Both are valid JSON, and neither is what the source held, and a strict diff will say so.

Six converters on 193,708 rows

The same file converted by six tools, with wall time and peak memory:

/usr/bin/time -f "%e s, peak %M KB" python lab/conv/j_py.py work/val/customers.csv work/conv/c_stream.json

1.87 s, peak 12944 KB

The streaming script used 13 MB and the others 16 to 40 times as much, and it is as fast as anything but DuckDB. csvjson is about twenty times slower than the Python and pandas scripts, which I put down to its type inference, though I did not profile it. DuckDB converted 193,708 rows in 0.38 s. Output sizes differ with the spacing (pandas writes no spaces after separators) and with type choices.

What each tool did to the values

Speed is secondary to whether the JSON still says what the CSV said. I read each output back and compared every value, as text, with the CSV:

ls -l work/conv/c_*.json | awk '{printf "%-18s %d bytes\n", $9, $5}' | sed 's#work/conv/##'
python - <<'PY'
import csv, json
orig = [dict(r) for r in csv.DictReader(open('work/val/customers.csv', newline='', encoding='utf-8'))]
for f in ['c_stream', 'c_list', 'c_pandas', 'c_duck', 'c_mlr', 'c_csvjson']:
    d = json.load(open('work/conv/' + f + '.json', encoding='utf-8'))
    same = len(d) == len(orig) and all({k: ('' if v is None else str(v)) for k, v in a.items()} == b for a, b in zip(d, orig))
    print(f"{f:10} {len(d)} objects; values equal to the CSV text: {same}; first id type: {type(d[0]['id']).__name__}")
PY

c_csvjson.json 52828897 bytes
c_duck.json 49111187 bytes
c_list.json 52745596 bytes
c_mlr.json 56182142 bytes
c_pandas.json 48096605 bytes
c_stream.json 52745597 bytes
c_stream   193708 objects; values equal to the CSV text: True; first id type: str
c_list     193708 objects; values equal to the CSV text: True; first id type: str
c_pandas   193708 objects; values equal to the CSV text: True; first id type: str
c_duck     193708 objects; values equal to the CSV text: True; first id type: str
c_mlr      193708 objects; values equal to the CSV text: False; first id type: int
c_csvjson  193708 objects; values equal to the CSV text: False; first id type: float

Four of the six matched the CSV text exactly: the two Python scripts, pandas with dtype=str, and DuckDB with all_varchar=true. Two did not, because they infer types:

head -n 2 work/val/customers.csv | tail -n 1
for f in c_stream c_duck c_mlr c_csvjson; do printf '%-10s' $f; python -c "
import json; d = json.load(open('work/conv/' + '$f' + '.json')); print(json.dumps(d[0], ensure_ascii=False)[:230])"; done

1,Cy,Riga,[email protected],+14192381408,,2025-04-03,10485.58,USD,4156,IT,active
c_stream  {"id": "1", "first_name": "Cy", "last_name": "Riga", "email": "[email protected]", "phone": "+14192381408", "phone_unparsed": "", "signup_date": "2025-04-03", "balance": "10485.58", "balance_currency": "USD", "units": "4156", "co
c_duck    {"id": "1", "first_name": "Cy", "last_name": "Riga", "email": "[email protected]", "phone": "+14192381408", "phone_unparsed": null, "signup_date": "2025-04-03", "balance": "10485.58", "balance_currency": "USD", "units": "4156", "
c_mlr     {"id": 1, "first_name": "Cy", "last_name": "Riga", "email": "[email protected]", "phone": 14192381408, "phone_unparsed": "", "signup_date": "2025-04-03", "balance": 10485.58, "balance_currency": "USD", "units": 4156, "country": "
c_csvjson {"id": 1.0, "first_name": "Cy", "last_name": "Riga", "email": "[email protected]", "phone": 14192381408.0, "phone_unparsed": null, "signup_date": "2025-04-03", "balance": 10485.58, "balance_currency": "USD", "units": 4156.0, "cou

The CSV row is 1,Cy,Riga,...,+14192381408,,2025-04-03,10485.58,USD,4156,.... Python wrote the phone as the string "+14192381408". **Miller wrote "phone": 14192381408**, a number, dropping the plus sign, and wrote id, balance and units as numbers. **csvjson wrote "id": 1.0, "phone": 14192381408.0 and "units": 4156.0**, floats, with the empty phone_unparsed as null. DuckDB (reading as text) wrote the empty phone_unparsed as null too. For a file of identifiers, both inferring tools changed data. For a file of measurements, Miller's output may be what you want. Both have a switch that turns inference off (Miller's -S and csvkit's -I), and with it each wrote "phone": "+14192381408" as a string, as I checked. I recommend turning inference off first and typing deliberately afterwards.

How to tell it worked

Read the JSON back and compare every value, as text, with the CSV, the way the last test did. Check the count of objects against the count of records (not lines). Look at one row with each awkward feature: the leading zero, the empty field, the newline, the non-ASCII text and a number with a trailing zero. If a consumer will parse the JSON, run it through that consumer's parser, since a double-precision parser is where precision is lost. And keep the conversion script with the data, so the decisions are written down.

When the numbers must be exact

If the values are money, measurements with a fixed scale or identifiers, put them in strings and let the consumer convert with its own decimal type. If the consumer needs JSON numbers, state the precision you can promise. If the CSV holds a hierarchy flattened into columns (address.city), the conversion also has to rebuild the nesting, which is a design step this article does not cover; the reverse direction, JSON to CSV with nested data, has that part. And for very large files, the next step is JSON Lines, which lets a consumer read one object at a time. The conversion hub compares all the targets.