Using the Python csv Module for Reading Files Line by Line

The csv module is in the standard library, has no dependencies, and reads a file one record at a time without holding the rest in memory. That makes it the right tool for a lot of small jobs: a one-off script, an import that runs inside a service, a transform that has to work on a server where you cannot install anything. It is also easy to use slightly wrongly in ways that work on the first file and fail on the tenth. This article covers the reader: how to open the file, what comes back, the few parameters that matter, and how it behaves on a 2,000,000 row file. The whole subject, including pandas and writing, is in the guide to reading and writing CSV in Python.

The smallest working reader

A file with a quoted comma, a doubled quote, a line break inside a field and an empty field:

id,name,city,note
1,Ana Silva,Lisbon,likes tea
2,"Smith, Jane",Boston,"said ""hello"""
3,Bo Chen,Oslo,"line one
line two"
4,Cy Dubois,Lyon,

The reader takes any iterable of lines, and a file object is one:

import csv

with open("people.csv", newline="", encoding="utf-8") as f:
    reader = csv.reader(f)
    header = next(reader)
    print(header)
    for row in reader:
        print(reader.line_num, row)

python /batches/csv-0001-0200/lab/py/read_basic.py

['id', 'name', 'city', 'note']
2 ['1', 'Ana Silva', 'Lisbon', 'likes tea']
3 ['2', 'Smith, Jane', 'Boston', 'said "hello"']
5 ['3', 'Bo Chen', 'Oslo', 'line one\nline two']
6 ['4', 'Cy Dubois', 'Lyon', '']

Every value is a string: '1', not 1. The header came off with next(reader), and the loop saw four records. The second record has Smith, Jane as a single value, quotes removed. The third has said "hello", the doubled quote collapsed to one. The fourth is line one\nline two, one value that contains a newline. The last has an empty string for the missing note, which is not None.

reader.line_num is the number of lines read from the source so far, not the number of records. It prints 2, 3, 5 and 6: the record for Bo Chen spans lines 4 and 5, so the counter jumps by two. Use it in error messages to point someone at the right place in the file, and keep your own counter if you need a record number.

Why newline="" is not optional

The documentation says to open CSV files with newline="", and the reason is concrete. By default, Python's text mode translates line endings: \r\n, \r and \n all arrive as \n. That is fine until a quoted field contains a line break, which is data and not structure. I made a file with Windows line endings and a field containing \r\n, then read it both ways:

import csv

for label, kwargs in [("newline=''", {"newline": ""}), ("default   ", {})]:
    with open("people-crlf.csv", encoding="utf-8", **kwargs) as f:
        rows = list(csv.reader(f))
    print(label, repr(rows[2][3]))

python /batches/csv-0001-0200/lab/py/crlf_open.py

newline='' 'line one\r\nline two'
default    'line one\nline two'

With newline="" the value is 'line one\r\nline two', exactly what the file holds. With the default it is 'line one\nline two'. Nothing raised and nothing warned. With newline="" the csv module sees the raw line endings and does its own handling, including the distinction between a record terminator and a newline inside quotes. The same rule applies when writing, with a worse symptom, which is the subject of the Windows blank line article. Also pass encoding every time. The default is the platform's locale encoding, which differs between machines, and a script that works on a Linux server with UTF-8 can fail on a Windows laptop with a different default. The encoding questions are covered in the garbled text guide.

The parameters that matter

The reader's dialect is a set of parameters, all keyword arguments. These are the ones I use:

import csv, io

text = "id;name;note\n1;Ana;'a;b'\n2;Bo;plain\n"
print(list(csv.reader(io.StringIO(text), delimiter=";")))
print(list(csv.reader(io.StringIO(text), delimiter=";", quotechar="'")))
print(list(csv.reader(io.StringIO("a, b, c\n"), skipinitialspace=True)))
print(list(csv.reader(io.StringIO("a, b, c\n"))))
try:
    list(csv.reader(io.StringIO('a,"b"c,d\n'), strict=True))
except csv.Error as e:
    print("strict:", e)
print(list(csv.reader(io.StringIO('a,"b"c,d\n'))))

python /batches/csv-0001-0200/lab/py/read_opts.py

[['id', 'name', 'note'], ['1', 'Ana', "'a", "b'"], ['2', 'Bo', 'plain']]
[['id', 'name', 'note'], ['1', 'Ana', 'a;b'], ['2', 'Bo', 'plain']]
[['a', 'b', 'c']]
[['a', ' b', ' c']]
strict: ',' expected after '"'
[['a', 'bc', 'd']]

The first two lines parse 1;Ana;'a;b' twice. With only delimiter=";", the apostrophes mean nothing and the record splits into "'a" and "b'": four fields, where three were meant. With quotechar="'" the same text becomes 'a;b', one field. The quote character is part of the dialect and the default " does not apply to files that quote with something else. The file's author decides this, not the reader; look at a few lines before choosing.

The third and fourth lines are a, b, c with and without skipinitialspace=True. The default keeps the spaces, ' b' and ' c'; the option drops the spaces that follow a delimiter. Use it for files written by hand, not for files from a program, in which a leading space could be data.

The last two lines are the one to know about. The text a,"b"c,d has a quote that closes before the field ends. By default the reader accepts it and returns bc, merging the quoted part and the text that follows it, with no error. With strict=True it raises csv.Error: ',' expected after '"'. A lenient parser is a good thing until you need to know the file is wrong, and for data you load into a system of record, I turn strictness on and let the error stop the job. The other dialect parameters, escapechar, doublequote and quoting, matter for unusual files and are described in the reference.

Streaming versus loading

The reader is an iterator, which means a loop over it holds one record at a time. To check what that is worth, I summed the amount column of the 2,000,000 row orders file in two ways. One sums over the reader directly. The other first calls list(reader):

import csv, sys, resource, time

mode = sys.argv[1]
t = time.perf_counter()
with open("/batches/csv-0001-0200/work/orders-2m.csv", newline="", encoding="utf-8") as f:
    reader = csv.reader(f)
    next(reader)
    if mode == "stream":
        total = sum(float(r[3]) for r in reader)
        n = reader.line_num - 1
    else:
        rows = list(reader)
        total = sum(float(r[3]) for r in rows)
        n = len(rows)
dt = time.perf_counter() - t
mb = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024
print(f"{mode:7s} rows={n} total={total:.2f} time={dt:.2f}s peak_rss={mb:.0f}MB")

python /batches/csv-0001-0200/lab/py/sum_stream.py stream; python /batches/csv-0001-0200/lab/py/sum_stream.py list

stream  rows=2000000 total=160174646.36 time=1.52s peak_rss=13MB
list    rows=2000000 total=160174646.36 time=3.92s peak_rss=1044MB

Both got 160,174,646.36. The streaming version took 1.52 s and its peak resident memory was 13 MB. The list version took 3.92 s and peaked at 1,044 MB, about 80 times more, for a 130 MB file, because every field is a separate Python string object with its own overhead. The time difference is mostly allocation. If you can process a row and move on, do not collect them. If you need several passes, write a first pass that keeps only the columns you need, or convert once to a format that loads faster, and see reading only some columns for the pandas equivalent.

Header handling and common mistakes

next(reader) returns the header, and raises StopIteration if the file is empty, so for a file that might be empty use next(reader, None). The reader does not check that every row has the header's length. A short row is a short list and an extra field is an extra element, so row[3] on a three field record raises IndexError, far from the cause. The inconsistent column counts article deals with this. If you prefer to address fields by name, csv.DictReader does that, and is the subject of its own article.

A blank line in the middle of a file is returned as an empty list, [], and not skipped. Test if not row: continue when you expect blank lines. The reader also never converts types: parse numbers and dates yourself, with float(row[3]) or datetime.strptime, and decide in one place what happens on a failure.

How to tell it worked

Count the records you read and compare with an independent count, such as the DuckDB or wc -l figures in the other articles for the same file; a file with embedded newlines will have more lines than records, which is exactly what line_num showed above. Check the length of every row against the header and report the line number of any that differ. Print the repr of one value that you know contains a comma, a quote or a newline. And run once with strict=True on a sample to see whether the file has quote problems that the default hides.

When csv.reader is not enough

If you need to select, filter, group and join, you will write a lot of loop code that a query would do better, and DuckDB or pandas is the faster route; the DuckDB guide has the comparison. If the file is damaged in ways the standard library cannot repair, such as mixed encodings or a quote that opens and never closes, no dialect option will fix it, and the repair has to happen before the reader sees the file. And if the file is too large for one process to parse in the time you have, csv is single threaded; the usual answer there is a different tool, not a cleverer loop.