Polars for Large CSV Processing Beyond Memory Limits

The usual complaint about processing a big CSV in Python is that the file does not fit in memory, or fits and makes the machine crawl. pandas reads the whole file into an in-memory table, and a 5 GB CSV needs more than 5 GB of RAM to do it. Polars approaches the problem differently: you describe the query lazily, and the engine reads and processes the file in batches, touching only the columns the query needs. This article runs the same group-by through pandas, eager Polars and lazy Polars on a 2 million row file, shows what the lazy API looks like, fixes the type inference trap that hits identifiers and ends with the cases where something else is the better choice. It is one of the tools surveyed in CSV tools worth knowing.

Lazy and eager

Polars has two modes. Eager mode, pl.read_csv(path), works like pandas: it reads the whole file into a DataFrame and each operation runs when you call it. Lazy mode, pl.scan_csv(path), returns a LazyFrame, a plan, and nothing is read until you call .collect(). The difference matters because the plan is optimized as a whole before it runs. A filter is pushed down to the reader so rows that do not match are discarded early, columns the query never mentions are not parsed at all, and the engine can process the file in chunks instead of holding it.

Here is the query used throughout: for shipped orders, count and total by country, and show the top three.

import polars as pl

lazy = (
    pl.scan_csv("orders-2m.csv", schema_overrides={"zip": pl.String})
    .filter(pl.col("status") == "shipped")
    .group_by("country")
    .agg(pl.len().alias("orders"), pl.col("amount").sum().round(2).alias("revenue"))
    .sort("revenue", descending=True)
    .head(3)
)
print(lazy.collect(engine="streaming"))

shape: (3, 3)
┌─────────┬────────┬──────────┐
│ country ┆ orders ┆ revenue  │
│ ---     ┆ ---    ┆ ---      │
│ str     ┆ u32    ┆ f64      │
╞═════════╪════════╪══════════╡
│ FR      ┆ 57651  ┆ 4.6162e6 │
│ IN      ┆ 57583  ┆ 4.6076e6 │
│ BR      ┆ 57184  ┆ 4.5940e6 │
└─────────┴────────┴──────────┘

This took 0.18 seconds in the run, with a peak resident memory of 266 MB for the process. The engine="streaming" argument asks for the batch-oriented engine, which processes the file in pieces. The default engine builds the result in memory, which is faster for small data and uses more memory on large data. In earlier releases the same switch was spelled streaming=True; check your version's documentation, since this API has moved between releases.

What it costs, side by side

To compare fairly, I ran one script three ways on the same file, a total of amount by status, and recorded the peak memory of each process:

pandas-all      0.98 s  peak RSS    484 MB
polars-eager    0.50 s  peak RSS    435 MB
polars-lazy     0.41 s  peak RSS    229 MB

The ordering is the point. pandas read all eight columns of the file and used 484 MB; eager Polars read all columns into Arrow-style columnar buffers and used 435 MB, half the time; lazy Polars saw that only status and amount were needed, read those two, streamed them and used 229 MB. The saving is proportional to how many columns you skip and how selective your filter is, and with a wide file it grows. For a file of 20 GB with 40 columns and a query touching four of them, the lazy plan is the difference between finishing and being killed by the operating system.

Do not over-read the memory column. It includes the Python interpreter and the libraries, and the floor is not zero. The conclusion that survives is that the lazy approach scales with the query, and the eager approaches scale with the file (how much memory pandas needs to load a CSV shows the rule of thumb for pandas).

Types: the zip code again

Polars infers each column's type from a sample of rows, 100 by default for read_csv and scan_csv via infer_schema_length. If the first rows have no leading zeros, a zip code column becomes an integer. In this file, the first rows have none, and later rows do:

df = pl.read_csv("orders-2m.csv", n_rows=5)
print(df.schema)
# Schema({'order_id': Int64, 'customer_id': Int64, 'zip': Int64, 'amount': Float64, ...})
df2 = pl.read_csv("orders-2m.csv", n_rows=5, schema_overrides={"zip": pl.String})
print(df2["zip"].to_list())
# ['99154', '34494', '93451', '19730', '67041']

zip is Int64 with the default and a list of strings with the override. The trap is that a later value such as 05398 parses perfectly well as the integer 5398, so nothing fails: the column loads, and the zero is gone from every code that had one (the two-row test with 02134 and 90210 gave [2134, 90210]). A parse error would at least be a warning. Do not raise infer_schema_length to the whole file as the cure, since that makes inference a full extra pass and still cannot tell you the column is an identifier. Set the types of identifier columns explicitly with schema_overrides, or read every column as text with infer_schema=False and cast the ones you need (why pandas turns zip codes into numbers describes the same trap in the other library).

Convert to Parquet while you scan

If you will query the data more than once, convert it. sink_parquet runs a lazy query and writes the result without holding it:

pl.scan_csv("orders-2m.csv", schema_overrides={"zip": pl.String}).sink_parquet("orders.parquet")

It took 0.29 seconds and produced a 34.1 MB file from the 123.9 MB CSV, with column types stored, so later reads do not infer anything. Parquet is columnar and compressed, and reading two columns of it touches a fraction of the bytes (why Parquet beats CSV for repeated analysis, converting CSV to Parquet). The sink approach is also the way to process a file that is larger than memory: scan, clean, write out, and work from the output.

Other things worth using

A few features matter on real files. pl.scan_csv("data/*.csv") reads a set of files as one frame, with the same schema logic. ignore_errors=True turns unparseable values into nulls, which hides bad data, so use it only when you also count the nulls. null_values=["NA", "-"] declares your null markers. try_parse_dates=True parses ISO dates, and explicit casts with pl.col("created_at").str.to_datetime("%Y-%m-%d %H:%M:%S") are the safer form. Group-by, joins, window functions and expressions compose; the expression API is larger than pandas' and worth an afternoon of reading. collect() on a frame that does not fit still fails, so keep the tail of a lazy query small (aggregate, filter, head) before collecting.

When pandas or DuckDB is the better fit

pandas is the right answer when the data is small enough, when a library you need expects a pandas frame (many plotting and machine learning tools do, though most now accept Polars through conversion), or when your team already knows it. The 2 million row file, in 0.98 seconds and 484 MB, was not a problem for it. The advantage of Polars appears at tens of millions of rows or a few gigabytes.

DuckDB answers the same question in SQL, and in my timing a little faster: the group-by on this file took 0.20 seconds from a cold start, with no Python at all (querying a CSV file with DuckDB instead of loading it). If your team thinks in SQL, if the question includes joins across several files or window functions, DuckDB may be the simpler choice, and the two interoperate: DuckDB can query a Polars frame directly and return a result as one. Pick on what your people read fluently, then measure on your own data (benchmarking CSV loading with pandas and DuckDB). The Polars documentation for the CSV functions is at docs.pola.rs, and the pandas scaling guide lists the options for large data in the older library.