How Much Memory pandas Needs to Load a CSV File in 2026

"Will this file fit?" is the first question about a big CSV and the one people answer worst. They look at the file size, guess, and find out when the process is killed. The answer depends on the number of rows, the types of the columns and the version of pandas, and it can be measured in a minute on a sample. This article reports what a 1.3 GB, 20,000,000-row file needed, shows how to measure your own, and gives a rule of thumb with its limits.

Two numbers, not one

Memory has a peak and a resident size. The resident size is what the DataFrame occupies once loaded. The peak is higher, because while the parser builds the columns it holds buffers, intermediate arrays and sometimes both the raw and converted data. When a job dies, it is the peak that killed it.

For the whole file with defaults:

The default load peaked at about three times the file and held about one and a half times. Typed, the same data held 0.72 GB. Two columns held 0.18 GB.

Where the memory goes

df = pd.read_csv("orders.csv")
print(df.memory_usage(deep=True) / 1e6)   # megabytes per column
print(df.dtypes)

deep=True matters: without it, pandas counts a text column as pointers and reports far less than the truth. Numeric columns are cheap and predictable: an int64 is 8 bytes a row, so 20,000,000 rows is 160 MB per column. Text is the cost. In the file above, created_at, status, country and note are text, and together they made up most of the 1.94 GB, because every row carries its own string. The levers:

• **category** for columns with few distinct values. status has five, so 20 million strings become 20 million small integers plus a five-entry table.

• Smaller numbers. int32 instead of int64 halves a column; float32 halves a float, at the cost of precision (about seven digits, unfit for money).

• **usecols** to skip columns you will not use. The parser does not build them at all.

• Parsing dates only when needed. parse_dates took the typed read from a few seconds to 13.47 s here; a timestamp kept as text costs memory but not time.

A zip that lost its zeros

The default read printed 'zip': 'int64'. The column holds five-digit strings, many with a leading zero, and pandas read them as integers: 09587 became 9587, silently. This is a memory topic because it is the same decision, a type chosen by inference: tell pandas what each column is (dtype={"zip": "str"}) and you control both the bytes and the meaning. The chunked reading article shows the full set of options.

Estimating before you load

You do not need to load the file to know what it will need. Read a sample, measure the bytes per row and multiply:

sample = pd.read_csv("orders.csv", nrows=200_000)
per_row = sample.memory_usage(deep=True).sum() / len(sample)
print(per_row * 20_000_000 / 1e9, "GB resident, about", 2 * per_row * 20_000_000 / 1e9, "GB peak")

The estimate is only as good as the sample. The first rows of a file are not typical (ids are shorter, early dates differ), as the row-counting article showed: an estimate from the head was 4% to 8% off. Sample from several places, or apply a margin of 25%. The factor of two for the peak is a guess that held in this test (the measured ratio was about 2 to 3 against resident); treat it as a starting point and test.

A rule of thumb, and where it breaks

For a typical mix of numbers and short text, with defaults: resident is 1.5 to 2 times the file size, peak is about three times. For mostly numeric data it is lower, since a number stored in 8 bytes is often 8 to 12 characters in the file. For text-heavy data, with long notes or many distinct strings, it is higher, because each Python string carries overhead. For a file of repeated categorical values with typed columns it can fall below the file size.

The rule breaks if the file has very wide columns (a free-text column of kilobytes), very many columns, or a lot of missing data in integer columns (which pandas converts to floats unless you ask for a nullable type). It also differs between pandas versions: pandas 3 stores strings with a compact representation, and older versions with Python objects use considerably more.

A worked estimate

Suppose a colleague sends a 6 GB CSV and you have a 16 GB laptop. The rule says a default load needs about 18 GB at peak and will fail. Reading only the five columns you need, with categories for the repeated text, might hold 1 GB resident and peak around 2 to 3 GB, which fits. The way to find out is the sample estimate above: read 200,000 rows with the exact usecols and dtype you intend to use, multiply the bytes per row by the row count, and add the margin. If the answer is under half your free memory, go ahead. If it is close, chunk. The estimate takes a minute and prevents a half-hour load that ends in a killed process.

Remember what else is running. The 13 GB container in this test had about 7 GB available while the file sat in the page cache, and a notebook that already holds three DataFrames has less than it appears to. When memory is short, the operating system starts swapping, the job does not fail but becomes ten times slower, and the symptom looks like a slow disk. Watching resident memory during the load, not only after it, tells you which case you are in.

How to tell it worked

Measure the peak, not the size: run the load under /usr/bin/time -v and read "Maximum resident set size". Compare with the estimate. Check df.dtypes for surprises. Then rerun with usecols and dtype and compare. If you are inside a container or a job scheduler with a memory limit, test inside it, because the limit applies to the peak.

When the rule of thumb fails

If the numbers say it will not fit, you have four ways out: read fewer columns, use smaller types, read in chunks and aggregate (the chunk article), or move the work to a tool that streams, such as DuckDB, which ran a grouped aggregate over the same file in 1.23 s with a 718,120 KB peak (DuckDB over CSV). The large-file hub compares them, and the loading benchmark puts time and memory side by side.