Reading CSV Files in Bash Without Breaking on Quotes

Every few months someone asks how to loop over the rows of a CSV in bash, and the answer in every short tutorial is while IFS=, read -r a b c; do ...; done < file.csv. It works on the example in the tutorial. It also silently mishandles quoted commas, a missing final newline, backslashes, Windows line endings and every field with a line break in it. This article shows each failure with the output, explains the small number of shell rules behind them, and gives a pattern that lets a real CSV parser do the splitting while bash keeps the loop. It is one of the cases in the tour of working with CSV in the shell.

The loop everyone writes

bash --version | head -n 1; mlr --version; qsv --version | cut -d- -f1; jq --version

GNU bash, version 5.2.15(1)-release (aarch64-unknown-linux-gnu)
mlr 6.22.0
qsv 26.0.0
jq-1.6

The loop splits each line on commas into named variables:

while IFS=, read -r id name city note; do echo "[$id] [$name] [$city] [$note]"; done < data/people.csv

[id] [name] [city] [note]
[1] [Ana Silva] [Lisbon] [likes tea]
[2] ["Smith] [ Jane"] [Boston,"said ""hello"""]
[3] [Bo Chen] [Oslo] ["line one]
[line two"] [] [] []
[4] [Cy Dubois] [Lyon] []

Records 1 and 4 are right. Record 2 is wrong: name is "Smith, city is Jane", and the third variable note collected everything else, Boston,"said ""hello""", because read puts the remainder of the line in the last variable. Record 3 is worse. The note contains a line break, so the loop saw two lines: the first has ["line one in the last field, and the second, line two", became a record of its own with the other three variables empty. A loop that counts records, inserts rows or builds a report from this has made up a record. Nothing failed.

This is the same shape of bug as with cut and awk: the shell's tools split on the delimiter and know nothing about quoting. Before the CSV-specific part, there are three smaller traps in the loop itself.

read -r, and the last line

read without -r treats a backslash as an escape and removes it:

printf 'id,path\n1,C:\\temp\\new\n' > /tmp/bs.csv
echo "--- read without -r:"; while IFS=, read id path; do echo "$path"; done < /tmp/bs.csv
echo "--- read -r:"; while IFS=, read -r id path; do echo "$path"; done < /tmp/bs.csv

--- read without -r:
path
C:tempnew
--- read -r:
path
C:\temp\new

For the line 1,C:\temp\new, plain read printed C:tempnew, and read -r printed C:\temp\new. Always use -r. The second trap is the final line. If the file does not end with a newline, read returns a non-zero status on the last line, because it hit end-of-file before a newline, and the while loop stops without processing it:

printf 'id,qty\n1,3\n2,5' > /tmp/nonl.csv
echo "--- plain loop:"; n=0; while IFS=, read -r id qty; do n=$((n+1)); done < /tmp/nonl.csv; echo "read $n lines"
echo "--- guarded:";  n=0; while IFS=, read -r id qty || [[ -n $id ]]; do n=$((n+1)); done < /tmp/nonl.csv; echo "read $n lines"

--- plain loop:
read 2 lines
--- guarded:
read 3 lines

The plain loop read 2 of 3 lines; with || [[ -n $id ]] added to the condition, which processes a final unterminated line when read filled something, it read 3. Exports from some systems omit the last newline, and a loop that drops the last record is a bug that sleeps for months.

What IFS does

The IFS variable decides how read splits. It treats two kinds of characters differently: whitespace characters (space, tab, newline) are collapsed and trimmed, and other characters are strict separators. That matters for empty fields:

printf '1\t\tc\n' | { IFS=$'\t' read -r a b c; echo "tab IFS:   a=[$a] b=[$b] c=[$c]"; }
printf '1,,c\n'  | { IFS=','   read -r a b c; echo "comma IFS: a=[$a] b=[$b] c=[$c]"; }
printf '1\x1f\x1fc\n' | { IFS=$'\x1f' read -r a b c; echo "US IFS:    a=[$a] b=[$b] c=[$c]"; }

tab IFS:   a=[1] b=[c] c=[]
comma IFS: a=[1] b=[] c=[c]
US IFS:    a=[1] b=[] c=[c]

With IFS=$'\t', the line 1<TAB><TAB>c gave a=[1] b=[c] c=[]: the two tabs counted as one separator, the empty middle field vanished, and c moved into b. With IFS=',', the same shape gave b=[] c=[c] and the empty field was kept. A non-whitespace separator, like the unit separator \x1f in the third line, also keeps empties. So if you convert a CSV to TSV and read it with IFS=$'\t', every record with an empty field is read wrong. It is an easy mistake to make when converting to TSV and then looping, and the reason the pattern below swaps the tab for another character.

Whitespace in IFS=',' is also not trimmed:

printf 'a, b ,c\n' | { IFS=, read -r x y z; echo "[$x] [$y] [$z]"; }
printf 'a,,c\n' | { IFS=, read -r x y z; echo "[$x] [$y] [$z]"; }
printf 'a b  c\n' | { IFS=' ' read -r x y z; echo "[$x] [$y] [$z]"; }

[a] [ b ] [c]
[a] [] [c]
[a] [b] [c]

a, b ,c gave [a] [ b ] [c], spaces kept, which is what you want from a CSV. A space-separated line with IFS=' ' collapsed a b c to three fields; for CSV the comma is the right separator and spaces are data.

Let a parser split, bash loops

The fix for quotes and line breaks is to give the loop something that is safe to split. Miller's TSV writer escapes the characters that would break a line-oriented reader: a real tab inside a value is written \t, and a line break is written \n.

mlr --icsv --otsv cat data/people.csv

id	name	city	note
1	Ana Silva	Lisbon	likes tea
2	Smith, Jane	Boston	said "hello"
3	Bo Chen	Oslo	line one\nline two
4	Cy Dubois	Lyon	

The people file came out as TSV. Smith, Jane is plain, and the note with the line break is on one line as line one\nline two, with a backslash and an n. Every record is now one line and every real tab is a separator. To avoid the whitespace collapse, translate those tabs to the unit separator, \037, which is not whitespace and cannot occur in the data after escaping:

mlr --icsv --otsv --headerless-tsv-output cat data/people.csv | tr '\t' '\037' | od -c | head -n 8

0000000   1 037   A   n   a       S   i   l   v   a 037   L   i   s   b
0000020   o   n 037   l   i   k   e   s       t   e   a  \n   2 037   S
0000040   m   i   t   h   ,       J   a   n   e 037   B   o   s   t   o
0000060   n 037   s   a   i   d       "   h   e   l   l   o   "  \n   3
0000100 037   B   o       C   h   e   n 037   O   s   l   o 037   l   i
0000120   n   e       o   n   e   \   n   l   i   n   e       t   w   o
0000140  \n   4 037   C   y       D   u   b   o   i   s 037   L   y   o
0000160   n 037  \n

The od dump shows 1 037 Ana Silva 037 Lisbon 037 likes tea \n: fields separated by octal 037 and records by newline, with the escaped \n of the note visible as two characters. The loop reads it:

while IFS=$'\x1f' read -r id name city note; do
  note=${note//\\n/$'\n'}
  printf '%s | %s | %s | %s\n' "$id" "$name" "$city" "$note"
done < <(mlr --icsv --otsv --headerless-tsv-output cat data/people.csv | tr '\t' '\037')

1 | Ana Silva | Lisbon | likes tea
2 | Smith, Jane | Boston | said "hello"
3 | Bo Chen | Oslo | line one
line two
4 | Cy Dubois | Lyon | 

Four records, each correct: Smith, Jane in one variable, said "hello" with its quotes, the empty note as an empty string, and for the third record the line break restored by ${note//\\n/$'\n'}, which replaces the two characters with a newline. Wrapped in a function, with the quantity column and a blank handled on purpose:

csv_rows() { mlr --icsv --otsv --headerless-tsv-output cat "$1" | tr '\t' '\037'; }
total=0; n=0; blank=0
while IFS=$'\x1f' read -r id customer city qty price status; do
  [[ -z $qty ]] && { blank=$((blank + 1)); qty=0; }
  n=$((n + 1)); total=$((total + qty))
done < <(csv_rows data/orders-small.csv)
echo "records: $n, total quantity: $total, blank quantities: $blank"

records: 8, total quantity: 40, blank quantities: 1

8 records, a total quantity of 40 and 1 blank quantity, matching what csvkit's csvstat reported for the same file. The pattern is csv_rows as the parser and read as the consumer. I used Miller (mlr --icsv --otsv); any tool that writes escaped TSV works the same way. A header can be read with the same trick, by dropping --headerless-tsv-output and consuming the first line.

JSON instead

If you already have jq, convert to JSON and let jq pull out fields. It is the safest of the options when the CSV has awkward values, since JSON has a complete escaping scheme:

mlr --icsv --ojsonl cat data/people.csv | jq -r '[.id, .name, .note] | @tsv'
echo ---
mlr --icsv --ojson cat data/people.csv | jq -r '.[] | select(.city == "Boston") | .name'

1	Ana Silva	likes tea
2	Smith, Jane	said "hello"
3	Bo Chen	line one\nline two
4	Cy Dubois	
---
Smith, Jane

jq -r '[.id, .name, .note] | @tsv' printed tab-separated lines with the line break as \n, and the second command selected a record by value (city == "Boston") and printed Smith, Jane. For anything that is a filter or a transform, one jq call over the whole stream is much faster than calling it per line; the pattern ... | while read -r line; do echo "$line" | jq ...; done starts a process per record.

Counting in bash

A loop can keep state. Counting records per city with an associative array:

declare -A seen
while IFS=$'\t' read -r id name city note; do seen[$city]=$((${seen[$city]:-0} + 1)); done < <(mlr --icsv --otsv --headerless-tsv-output cat data/people.csv)
for k in "${!seen[@]}"; do echo "$k ${seen[$k]}"; done | sort

Boston 1
Lisbon 1
Lyon 1
Oslo 1

Each of the four cities appeared once. declare -A needs bash 4 or later, which is the macOS caveat above. The loop is the right shape when the logic per record is more than a one-liner (call a command, branch on a value, write to several files), and the wrong shape when it is a sum or a group by.

Speed

Bash reads about 125,000 lines per second. On the 2,000,000 row file:

time (n=0; while IFS=, read -r a b c d e f g h; do n=$((n+1)); done < data/orders-2m.csv; echo "bash loop read $n lines")
time (qsv count data/orders-2m.csv)
time (mlr --icsv --otsv --headerless-tsv-output cat data/orders-2m.csv | wc -l)

real	0m16.137s
user	0m13.921s
sys	0m2.050s
bash loop read 2000001 lines
2000000

real	0m0.232s
user	0m0.141s
sys	0m0.027s
2000000

real	0m1.762s
user	0m3.092s
sys	0m0.751s

The plain read loop, which only counted lines, took 16.1 s. qsv count took 0.23 s and the Miller conversion to TSV, writing 2,000,000 lines, took 1.76 s. So the cost of the pattern is two stages: the parse is fast, the loop is slow. For a few hundred thousand records it is fine; for millions, move the logic out of the loop and into awk, mlr or SQL, and let the shell orchestrate. A bash loop that makes a call to an external command per record, such as date or curl, is slower again by orders of magnitude.

How to tell it worked

Count the iterations of the loop and compare with qsv count. Print one record with a comma in a name, one with a line break and one with an empty middle field. Test a file without a final newline and a file with a BOM or CRLF (strip them first: tr -d '\r' and sed '1s/^\xef\xbb\xbf//'). Run with set -euo pipefail and confirm a failing parser fails the script, as the shell guide shows.

When a bash loop is the wrong tool

If the work is arithmetic, grouping or filtering, use awk on unquoted data or Miller, qsv or DuckDB on anything else. If the loop calls other commands for each row, consider generating the commands with the parser and running them with xargs -P. And if the CSV is the input to a real program, write that program in a language with a CSV library, as in the other articles of this series.