How Many Rows Can Pandas Read From Excel Before It Struggles

People ask for a number: how many rows can pandas read from an Excel file? The answer sounds evasive and is accurate: it depends on how much memory you have, what is in the rows and how you read them. A sheet holds at most 1,048,576 rows, so the ceiling is never above that for one sheet, and it is often far below it. This post shows how to estimate the memory a DataFrame needs before you read it, why the peak is higher than the final size, how to reduce the load and what to do when the file will not fit. It is a spoke of Excel files that are too big or too slow.

There is no row limit, only memory

pandas.read_excel has no chunksize argument, unlike read_csv. It reads the sheet through its engine, builds the whole table and returns a DataFrame. Nothing caps the number of rows except the sheet's own limit (1,048,576) and the memory of the machine. When the read exceeds memory, you get a MemoryError, a killed process or a machine that swaps to disk and crawls.

So the useful question is: how much memory does this file need? Two parts: the final DataFrame and the peak during the read.

Estimate the final DataFrame

Size depends on column types.

A table of one million rows with five numeric columns is about 40 MB. Add five text columns as object and the number can jump to several hundred megabytes, since each cell is a Python string object. Strings dominate. A quick estimate for a table of n rows: n × (8 × numeric_columns + ~70 × text_columns) bytes for object strings, which is rough and conservative. Measure on a sample:

import pandas as pd

sample = pd.read_excel("big.xlsx", nrows=50_000)
per_row = sample.memory_usage(deep=True).sum() / len(sample)
print(f"{per_row:.0f} bytes per row; 1M rows ~ {per_row * 1_000_000 / 1e9:.2f} GB")

deep=True counts the actual size of Python string objects. Multiply by the row count in the full sheet (open the file in read-only mode and count rows, or ask for it). This gives the final DataFrame size.

Why the peak is higher

During the read, pandas does not hold only the finished frame. With the openpyxl engine, the parser holds the sheet's cell data while it converts it to Python lists, then builds columns, then infers types. At the peak you can hold the raw rows and the partially built frame at the same time. A common rule of thumb is that the peak is a multiple of the final size, and that multiple depends on the engine and the types. Do not plan to use the whole of your RAM for the final frame. If the final DataFrame would be half of your memory, expect trouble.

The file size on disk is a poor guide, as the compressed XML expands: see reading a 500 MB Excel file without running out of memory for why. A 50 MB xlsx may be around one million cells of text, or ten million cells of numbers; the memory differs.

Make the read cheaper

Several arguments reduce the load.

**usecols.** Read only the columns you need. Dropping half the columns roughly halves both time and memory. usecols="A:F" or a list of names:

df = pd.read_excel("big.xlsx", sheet_name="Data", usecols=["order_id", "order_date", "region", "amount"])

**dtype.** State types so pandas does not infer. Use smaller types where values allow (int32, float32), category for repeated text, and str for identifiers:

df = pd.read_excel("big.xlsx", usecols=["order_id", "region", "amount"],
                   dtype={"order_id": "string", "region": "category", "amount": "float32"})

A region column with five distinct values as category is one byte per row, instead of dozens. Reading identifiers as text keeps leading zeros (specifying column types when reading Excel).

**nrows and skiprows.** Read part of the sheet: nrows=100_000 for a sample, or skiprows and nrows to take a slice. Each call re-parses from the top of the sheet, so reading ten slices costs ten passes, not one.

**engine.** engine="calamine" (pandas 2.2 or later with python-calamine installed) parses faster and with less overhead than openpyxl. It does not return formulas or styles. The default openpyxl engine works for most files.

**dtype_backend="pyarrow".** Returns Arrow-backed columns, which store strings compactly. A good memory saver for text-heavy data on a recent pandas.

**sheet_name.** Read one sheet. sheet_name=None reads every sheet into a dictionary, which multiplies memory.

Read one sheet at a time

A workbook with several large sheets should be read sheet by sheet, processing and releasing each. Use pd.ExcelFile to avoid reopening the file for every sheet:

with pd.ExcelFile("big.xlsx", engine="openpyxl") as xl:
    for name in xl.sheet_names:
        df = xl.parse(name, usecols="A:H", dtype=str)
        write_out(name, df)
        del df

ExcelFile parses the workbook structure once. Each parse still reads that sheet fully, so only one sheet is in memory at a time.

When the file does not fit

Past a certain size, the answer is to change the format, not to squeeze the read. read_excel cannot stream. Options:

1. Convert to Parquet or CSV once with a streaming tool (openpyxl read-only mode, DuckDB or LibreOffice), then read the result in pandas. read_csv has chunksize, and read_parquet can select columns and row groups (converting a huge Excel file to CSV or Parquet).

2. Use DuckDB to query the workbook directly and return only the aggregate to pandas.

3. Use a streaming reader (openpyxl read_only=True) in a loop that processes batches.

4. Get more memory, or a bigger machine, if the job is one-off and you can.

Choose by whether the read repeats. A single read for a one-time answer can use more memory for an hour. A weekly job should convert once and stop reading Excel.

Check the result

After reading, compare the row count and a control total with the workbook. A silent nrows or a sheet that ended at the row limit gives fewer rows than you think. Print df.dtypes and df.memory_usage(deep=True).sum() and compare with your estimate. Look at the first and last rows, since a trailing totals row is a frequent surprise. Convert types once and report the values that failed.

Rules of thumb, stated carefully

I cannot give you a row count, and these are only the shape of an answer. On a machine with ordinary memory, a sheet with a few hundred thousand rows and a dozen columns of mixed types reads comfortably in pandas. A sheet near the million-row limit with many text columns is where the troubles start, and where usecols, dtype and a faster engine matter. A multi-sheet workbook of that size should be read one sheet at a time. Anything much beyond that belongs in another format. Test with nrows on your file, scale the estimate and decide.

When no pandas setting will make the file fit

The data is more than memory even after trimming. Aggregate outside pandas.

The reading happens on a small machine. A 512 MB container cannot read a big workbook; do the conversion elsewhere.

You need all the columns as strings. Text-heavy frames in object dtype are expensive; try Arrow strings or categories, or process in pieces after conversion.

The file is the wrong shape. A report layout with merged cells and notes needs cleaning before size matters (cleaning Excel data before analysis).

See the pandas reference for read_excel and its scaling to large datasets guide. To check a small extract as JSON, use the CSV to JSON converter.