Python CSV Error Field Larger Than Field Limit and the Fix

_csv.Error: field larger than field limit (131072) stops a Python script on a file that opens fine in a text editor. Every answer you will find tells you to call csv.field_size_limit with a bigger number, and for one of the two causes that is correct. For the other it makes the error go away while the file is still broken, and you end up with a single record that contains half of your data. This article reproduces both causes, shows how to tell them apart in a few lines, and gives the fix for each. The reader itself is described in the csv module article, and the other quoting failures in how CSV quoting works.

The two files

The first has one honest cell of 200,000 characters, a payload that a program stuffed into a column. The second has an ordinary table of about 5,000 rows, with a quote opened on row 2 and never closed:

ls -l bigcell.csv unterminated.csv | awk '{print $5, $9}'

200024 bigcell.csv
226618 unterminated.csv

bigcell.csv is 200,024 bytes and unterminated.csv is 226,618 bytes. The second file starts like this: id,name,note, then 1,Ana,fine, then 2,Bo,"oops this quote never closes, and then 4,997 well-formed rows that follow. Reading both:

import csv, sys

print("default limit:", csv.field_size_limit())
for path in ["bigcell.csv", "unterminated.csv"]:
    try:
        with open(path, newline="", encoding="utf-8") as f:
            n = sum(1 for _ in csv.reader(f))
        print(path, "rows:", n)
    except csv.Error as e:
        print(path, "->", type(e).__name__ + ":", e)

python /batches/csv-0001-0200/lab/py/field_limit.py

default limit: 131072
bigcell.csv -> Error: field larger than field limit (131072)
unterminated.csv -> Error: field larger than field limit (131072)

The default limit prints as 131072, which is 128 KiB. Both files fail with the same message, Error: field larger than field limit (131072), and nothing in the message says which of the two situations you are in. The module's name for the exception is _csv.Error, which prints as Error here.

Why an unclosed quote looks like a huge field

When the reader meets an opening quote, everything up to the matching closing quote is one field, including newlines and delimiters. If the closing quote never comes, the reader keeps consuming lines into the same field until it reaches the end of the file or the limit. That is the situation a limit guards against: a malformed file should not be able to make the process allocate gigabytes for one value. So this error is often the first symptom of a quoting problem, not of a large value, and the file in question is typically small.

That also explains a pattern: the file is 226 KB and the field is 131 KB. If the field limit is hit in a file whose total size is only slightly larger than the limit, or whose rows are all short, think unclosed quote first.

Telling them apart

Raise the limit enough to read the file, and measure how many lines each record spans. A legitimately large cell is one record on one line (or a few), while a runaway quote spans thousands:

import csv

csv.field_size_limit(1_000_000)
with open("unterminated.csv", newline="", encoding="utf-8") as f:
    r = csv.reader(f)
    header = next(r)
    prev = r.line_num
    for row in r:
        span = r.line_num - prev
        if span > 1:
            print(f"record starting at line {prev + 1} spans {span} lines and has {len(row)} fields")
        prev = r.line_num

python /batches/csv-0001-0200/lab/py/find_unbalanced.py

record starting at line 3 spans 4998 lines and has 3 fields

record starting at line 3 spans 4998 lines and has 3 fields. That is the damaged file's signature: one record containing almost the whole file. The big cell file would print nothing, since its record sits on a single line. Another quick test is strict=True, which turns the lenient parse into an error:

import csv

csv.field_size_limit(1_000_000)
with open("unterminated.csv", newline="", encoding="utf-8") as f:
    rows = list(csv.reader(f, quoting=csv.QUOTE_NONE))
print("QUOTE_NONE   rows:", len(rows), "| row 3:", rows[2])
try:
    with open("unterminated.csv", newline="", encoding="utf-8") as f:
        list(csv.reader(f, strict=True))
except csv.Error as e:
    print("strict=True  ->", e)

python /batches/csv-0001-0200/lab/py/field_limit_alt.py

QUOTE_NONE   rows: 5000 | row 3: ['2', 'Bo', '"oops this quote never closes']
strict=True  -> unexpected end of data

With a raised limit and strict=True, the reader reports unexpected end of data: the file ended inside a quoted field. That message names the real problem, which the field limit message does not. The first line of the output shows a last resort, quoting=csv.QUOTE_NONE, which treats quotes as ordinary characters. It read the damaged file as 5,000 records, and row 3 came out as ['2', 'Bo', '"oops this quote never closes'], with the stray quote preserved. That would break any legitimately quoted field in the same file, so it is a way of looking at the damage and not a fix for a file that also contains good quoted values.

Fix 1: a genuinely large field

If the cell really is that large, raise the limit before you read. field_size_limit returns the previous value, which is useful for logging:

import csv, sys

old = csv.field_size_limit(1_000_000)
print("raised from", old, "to", csv.field_size_limit())
for path in ["bigcell.csv", "unterminated.csv"]:
    with open(path, newline="", encoding="utf-8") as f:
        rows = list(csv.reader(f))
    longest = max(len(c) for r in rows for c in r)
    print(f"{path:18s} rows={len(rows):5d} longest field={longest}")
    if path == "unterminated.csv":
        print("   row 3 starts:", rows[2][2][:48].replace("\n", "\\n"), "...")
        print("   fields in that row:", len(rows[2]))

python /batches/csv-0001-0200/lab/py/field_limit_fix.py

raised from 131072 to 1000000
bigcell.csv        rows=    3 longest field=200000
unterminated.csv   rows=    3 longest field=226588
   row 3 starts: oops this quote never closes\n3,Name3,some ordina ...
   fields in that row: 3

It moved the limit from 131072 to 1,000,000, and bigcell.csv then loaded as 3 rows (header and two records) with a longest field of 200,000 characters. Pick a number that is generous for your data and finite; the limit is a safety valve, and sys.maxsize takes it off.

The same loop on unterminated.csv also succeeded: 3 rows, longest field 226,588, and the third row starts oops this quote never closes\n3,Name3,some ordina ... and has 3 fields. It "worked" and produced a wrong file with two rows. This is the danger of the one-line answer from the search results: the exception was the only warning you were going to get.

The largest limit

Some scripts set the limit to sys.maxsize. On this platform that is accepted, and the maximum value is 9,223,372,036,854,775,807. A value that is too large raises a different error:

import csv, sys

try:
    csv.field_size_limit(2**63)
except OverflowError as e:
    print("OverflowError:", e)

limit = sys.maxsize
while True:
    try:
        csv.field_size_limit(limit)
        break
    except OverflowError:
        limit //= 10
print("accepted limit:", csv.field_size_limit(), "| sys.maxsize:", sys.maxsize)

python /batches/csv-0001-0200/lab/py/field_limit_max.py

OverflowError: Python int too large to convert to C ssize_t
accepted limit: 9223372036854775807 | sys.maxsize: 9223372036854775807

csv.field_size_limit(2**63) raised OverflowError: Python int too large to convert to C ssize_t. The loop that divides by ten until the call succeeds is a common defensive idiom, and here it accepted the first value, sys.maxsize. The loop exists because on some platforms and versions the largest accepted value is smaller than sys.maxsize; I could only test Linux with Python 3.13, so I cannot say which. Keep it if the script runs on machines you do not control. If the field is only a few hundred kilobytes, a fixed limit of 10 or 100 MB is simpler and keeps the safety valve.

Fix 2: repair the file

If the diagnosis is an unclosed quote, the repair is in the file, not in the limit. Find the line where the runaway record starts (the script above prints it: line 3), look at it, and correct the source or the line. Typical causes are a quote in the data that the writer did not double (6" pipe), a field that was truncated by a length limit in the export, and a manually edited file. Once the line is fixed, the limit does not need to change. If you cannot fix the source and need to carry on, read the file with a parser that you control, as in the inconsistent column counts article, and quarantine the damaged records.

In pandas

pandas has its own parser, and the answer depends on the engine. The default C engine read bigcell.csv without complaint and reported field lengths of [200000, 5]. With engine="python" it raised ParserError: field larger than field limit (131072), because the Python engine uses the csv module underneath. So if the same file works in one pandas call and fails in another, check for engine="python", which is also what you need for a callable on_bad_lines. To make the Python engine accept a large cell, call csv.field_size_limit before the read; I did not test that route.

How to tell it worked

Count the records and compare with the number of lines you expect from another tool: for a file with no embedded newlines, wc -l less one is the number of records. Check the longest field, max(len(c) for r in rows for c in r), and confirm it is the one you expected. Run the width check from the column counts article, and make sure a single record does not span thousands of lines. And run once with strict=True as a lint; it is quiet on a good file.

When raising the limit hides the problem

If you cannot explain why the field is that long, do not raise the limit. A large field that makes sense, such as an embedded document, a base64 blob or a log message, deserves a limit chosen for it. A field that is large because the quote is wrong deserves a repair. And a file that generates this error every week deserves a conversation with the system that writes it, since a quoting bug at the source is cheaper to fix once than to detect forever.