Limiting File Size and Row Count for CSV Uploads Safely

Every upload form eventually gets the file that hurts: 2 GB, or 40 million rows of one column, or a header 100,000 fields wide. The usual response is to set a size limit of 10 MB or 50 MB because those are the numbers people reach for, and that does a third of the job. A byte limit says nothing about rows, and the number of rows is what decides how long a worker is busy. This article shows how to measure your own cost per row, turn it into limits you can defend, and enforce each limit where it is cheapest. It is the numbers companion to why a hostile CSV upload can hurt your application server, and part of the wider CSV security risks that catch teams out.

Five limits, not one

A file has more than one dimension, and each can be abused alone.

The byte limit belongs at the edge. In nginx it is client_max_body_size 50m;, in PHP upload_max_filesize and post_max_size, and every framework has its own. Rejecting there means the application never reads the body. The other four need the parser, because only the parser knows what a row is.

Measure what a row costs

The row cap should come from time, not from a feeling. The question is how long a worker can be held by one upload, and the answer is a number your team picks: say 60 seconds of work in a background job, with a timeout above that. Divide by the cost per row of everything you do to a row.

Here is the measurement, on the 1.1 million row file, for three levels of work:

import csv, sqlite3, sys, time

REGIONS = {"north", "south", "east", "west"}

def timed(label, fn):
    t = time.perf_counter()
    n = fn()
    print(f"{label:<28}{n:>10} rows {time.perf_counter() - t:6.2f} s")

def parse():
    with open(sys.argv[1], newline="") as f:
        return sum(1 for _ in csv.reader(f))

def validate():
    n = 0
    with open(sys.argv[1], newline="") as f:
        r = csv.reader(f)
        next(r)
        for rid, region, amount in r:
            if int(rid) < 1 or region not in REGIONS or not 0 <= int(amount) <= 1000:
                raise ValueError(rid)
            n += 1
    return n

def load():
    db = sqlite3.connect(":memory:")
    db.execute("CREATE TABLE t (id INTEGER, region TEXT, amount INTEGER)")
    with open(sys.argv[1], newline="") as f:
        r = csv.reader(f)
        next(r)
        db.executemany("INSERT INTO t VALUES (?,?,?)", r)
    return db.execute("SELECT count(*) FROM t").fetchone()[0]

timed("parse only", parse)
timed("parse + validate", validate)
timed("parse + insert (SQLite)", load)

parse only                     1100001 rows   0.30 s
parse + validate               1100000 rows   0.40 s
parse + insert (SQLite)        1100000 rows   0.92 s

Parsing alone ran at about 3.7 million rows per second, validation of three fields at 2.7 million and an in-memory insert at 1.2 million. With a network database, a transaction per row and a few indexes, the last number drops by a factor of 50 to 500, and a file that loads in a second in this test takes minutes in production. That is the reason a measured limit beats a copied one: the same file is a second of work in one pipeline and ten minutes in another.

Peak memory for the parse loop, measured with the system's time -l, was about 15 MB. The file is read as a stream, so the memory cost is one row, not one file. It only becomes the constraint when the code builds a list or a dataframe of the whole file before doing anything (reading a large CSV in chunks with pandas shows the bounded alternative).

From a cost to a cap

Take the slowest honest step in your pipeline and its measured rate. If a real load runs at 20,000 rows per second against your production database, a 60 second budget is 1.2 million rows. Round down, and then halve it, because your test file had three narrow columns and real files have forty wide ones. A file with 40 columns of text parses several times slower per row than the benchmark, and a validation step that checks every field against a pattern is slower still.

Then state the cap in terms the user understands: "Files up to 1,000,000 rows". A byte cap derived from it is a sanity check: 1,000,000 rows at a generous 500 bytes each is 500 MB, and a body limit of 50 MB means your row cap is never reached on wide files and sometimes reached on narrow ones, which is fine. Choose which one you want to be the binding limit, and set the other to be looser.

For columns, count from your real schemas. If your widest legitimate file has 45 columns, a cap of 60 leaves room and stops the 100,000-column header. For field length, use the longest legitimate value times a safety factor. A product description of 4,000 characters suggests a limit of 10,000.

Enforce shape in the parser

Here is a function that enforces the three shape limits and says where it stopped:

import csv, sys

MAX_ROWS = 5_000_000
MAX_COLUMNS = 60
MAX_FIELD = 10_000

class Rejected(Exception):
    pass

def check_shape(path):
    csv.field_size_limit(MAX_FIELD)
    rows = 0
    with open(path, newline="", encoding="utf-8") as f:
        reader = csv.reader(f)
        try:
            header = next(reader)
            if len(header) > MAX_COLUMNS:
                raise Rejected(f"{len(header)} columns in the header, limit is {MAX_COLUMNS}")
            for rows, row in enumerate(reader, start=1):
                if len(row) != len(header):
                    raise Rejected(f"line {reader.line_num} has {len(row)} fields, expected {len(header)}")
                if rows > MAX_ROWS:
                    raise Rejected(f"more than {MAX_ROWS} rows")
        except csv.Error as e:
            raise Rejected(f"line {reader.line_num}: {e}") from e
    return rows

for path in sys.argv[1:]:
    try:
        print(path, "ok,", check_shape(path), "rows")
    except Rejected as e:
        print(path, "rejected:", e)

Run on four small files, one good and three bad:

ok.csv ok, 2 rows
wide.csv rejected: 80 columns in the header, limit is 60
longfield.csv rejected: line 2: field larger than field limit (10000)
ragged.csv rejected: line 3 has 1 fields, expected 2

Each rejection names the cause and a line, which is what the person who uploaded the file needs to fix it. A bare "invalid file" sends them to support. The ragged check doubles as a validity test, because a row with the wrong number of fields means a broken quote or a stray delimiter (checking row and column counts in a CSV). Run the shape check as a separate pass before loading, or inside the load loop if you prefer a single pass and can roll back.

Report limits before the upload

A limit that surprises people at the end of a 400 MB upload is a poor limit. State the caps on the upload page, check the declared size in the browser before sending (a File.size comparison takes one line), and return a clear error code when the server refuses. Log every rejection with the account, the size and the reason; a spike in rejections is the earliest sign of someone probing the endpoint, and a steady trickle from one customer is a sign that your limit is too low for a legitimate use. If you stream the file to storage and process it in a queue, enforce the limits again in the worker, since the web tier cannot see rows.

When the limits should be higher

Raise the caps when the uploader is a trusted internal process and the work is meant to be large. A migration of a 3 GB archive is not an attack, and refusing it teaches people to bypass the endpoint. The remedy is a separate path: an authenticated internal tool, a command-line loader that reads from disk with its own, higher limits, or a bulk interface that streams into a staging table (loading a CSV into a staging table, then merging).

Limits also need revisiting when your data changes. A file format that grows one column per quarter will outgrow a cap of 60, and a new customer with a hundred times the volume needs a conversation, not a silent failure. Put the numbers in configuration, review them when a rejection is reported, and keep the benchmark script in the repository so the next person can rerun it on their hardware. The parser settings used above are documented in Python's csv module reference, and OWASP's file upload cheat sheet lists the surrounding controls.