Reading and Writing CSV in Go With encoding csv and Streams

Go's standard library has a CSV package, encoding/csv, and for most jobs it is all you need. It follows the format closely, it is a streaming reader by design, and its errors tell you a line number. What it does not do is stranger than what it does: it does not strip a byte order mark, it does not convert types, and it decides the width of every record from the first one unless you say otherwise. This article reads and writes the same small files as the rest of this series, measures the streaming and the load-it-all forms on 2,000,000 rows, and lists the options worth knowing. The comparison with other languages is in the guide to reading CSV in Go, Java, R, Rust and C#.

Reading records

csv.NewReader takes any io.Reader. Read returns one record as a []string, and io.EOF when the file ends:

func people(path string) {
	r := csv.NewReader(open(path))
	for {
		rec, err := r.Read()
		if err == io.EOF {
			break
		}
		if err != nil {
			fmt.Println("error:", err)
			return
		}
		line, col := r.FieldPos(0)
		fmt.Printf("line %d col %d: %q\n", line, col, rec)
	}
}

cd go && go run . people ../data/people.csv

line 1 col 1: ["id" "name" "city" "note"]
line 2 col 1: ["1" "Ana Silva" "Lisbon" "likes tea"]
line 3 col 1: ["2" "Smith, Jane" "Boston" "said \"hello\""]
line 4 col 1: ["3" "Bo Chen" "Oslo" "line one\nline two"]
line 6 col 1: ["4" "Cy Dubois" "Lyon" ""]

Each record is a slice of strings. "Smith, Jane" is one field with the quotes removed, "said \"hello\"" has its doubled quotes collapsed, and the note with a line break is "line one\nline two". FieldPos(0) returns the line and column where field 0 started, so the records began on lines 1, 2, 3, 4 and 6: the fourth record spans lines 4 and 5, and the next starts at 6. That is the number to put in an error message.

Every value is a string. Converting is your job, with strconv, and so is deciding what an empty string means. For leading zeros that is an advantage: a ZIP code is a string until you decide otherwise.

The field count

Unless you change it, the reader takes the number of fields in the first record and requires every later record to match. A mismatch is an error, but not a fatal one: Read returns the record along with the error, and the next call continues. On the ragged test file, with a long record, a trailing comma, a short record and a record broken across two lines:

func widths(path string, relax bool) {
	r := csv.NewReader(open(path))
	if relax {
		r.FieldsPerRecord = -1
	}
	n := 0
	for {
		rec, err := r.Read()
		if err == io.EOF {
			break
		}
		if err != nil {
			var pe *csv.ParseError
			if errors.As(err, &pe) {
				fmt.Printf("ParseError line %d column %d: %v (is ErrFieldCount: %v)\n", pe.Line, pe.Column, pe.Err, errors.Is(err, csv.ErrFieldCount))
			} else {
				fmt.Println("error:", err)
			}
			if !relax {
				continue
			}
			return
		}
		n++
		_ = rec
	}
	fmt.Println("records read:", n)
}

cd go && go run . widths ../data/ragged2.csv

ParseError line 3 column 1: wrong number of fields (is ErrFieldCount: true)
ParseError line 5 column 1: wrong number of fields (is ErrFieldCount: true)
ParseError line 6 column 1: wrong number of fields (is ErrFieldCount: true)
ParseError line 8 column 1: wrong number of fields (is ErrFieldCount: true)
records read: 5

cd go && go run . widths ../data/ragged2.csv relax

records read: 9

The strict run reported four ParseErrors, on lines 3, 5, 6 and 8, each satisfying errors.Is(err, csv.ErrFieldCount), and counted 5 good records. Setting FieldsPerRecord = -1 turned the check off, and all 9 records were returned, including the bad ones, with their real lengths. There is a third setting, a positive number, which requires exactly that many fields.

Look at the choice this gives you. A loop that does if err != nil { return err } stops at the first bad record, which is safe. A loop that checks for ErrFieldCount, logs the line and carries on is a lossy import, and the log is the only evidence. I prefer to collect the bad lines in a slice and fail at the end if there are any, so that one run reports all of them.

Bad quotes

The package is strict about quoting, as described in how CSV quoting works. A quote in the middle of an unquoted field is an error, and LazyQuotes relaxes it:

func lazy() {
	for _, lazy := range []bool{false, true} {
		r := csv.NewReader(stringsReader("a,say \"hi\" now,c\n"))
		r.LazyQuotes = lazy
		rec, err := r.Read()
		fmt.Printf("LazyQuotes=%v -> %q err=%v\n", lazy, rec, err)
	}
}

cd go && go run . lazy

LazyQuotes=false -> ["a"] err=parse error on line 1, column 7: bare " in non-quoted-field
LazyQuotes=true -> ["a" "say \"hi\" now" "c"] err=<nil>

With the default, a,say "hi" now,c failed with parse error on line 1, column 7: bare " in non-quoted-field and returned only ["a"]. With LazyQuotes = true it returned all three fields, say "hi" now with its quotes intact. Use it for a producer that writes quotes carelessly, and count the cases.

A quote that opens and never closes is reported with both lines:

cd go && go run . people ../data/unterminated.csv | tail -n 2

line 2 col 1: ["1" "Ana" "fine"]
error: record on line 3; parse error on line 5000, column 47: extraneous or missing " in quoted-field

The error read record on line 3; parse error on line 5000, column 47: extraneous or missing " in quoted-field. The record began on line 3, where the stray quote is, and the parser gave up on line 5000. That is more useful than the "somewhere before here" of the Node parser, since the first number is where to look. Go, Java, Ruby and Kotlin all report this file as an error; Rust, C# and PHP returned the rest of it as one giant field with no complaint.

The BOM

encoding/csv does not interpret a byte order mark. A file saved by Excel starts with EF BB BF (the BOM article explains where it comes from), which is decoded as the character U+FEFF and becomes part of the first header:

func bom(path string) {
	r := csv.NewReader(open(path))
	rec, _ := r.Read()
	fmt.Printf("header with BOM left in: %q\n", rec)
	f := open(path)
	br := bufio.NewReader(f)
	if b, err := br.Peek(3); err == nil && b[0] == 0xEF && b[1] == 0xBB && b[2] == 0xBF {
		br.Discard(3)
	}
	rec, _ = csv.NewReader(br).Read()
	fmt.Printf("header after discarding the BOM: %q\n", rec)
}

cd go && go run . bom ../data/bom.csv

header with BOM left in: ["\ufeffid" "name"]
header after discarding the BOM: ["id" "name"]

The first call printed ["id" "name"]. The fix is to peek at the first three bytes through a bufio.Reader and discard them if they match, as the second half of the function does: ["id" "name"]. Make that a small helper and use it on every file that came from a person. The same goes for other encodings. The reader assumes UTF-8, and a file in Windows-1252 needs golang.org/x/text/encoding/charmap or a conversion before the reader sees the bytes.

Streaming and memory

Read allocates a new slice for each record by default. ReuseRecord = true reuses the same backing array, which removes most of the garbage when you only look at a record and move on. With a 1 MB bufio.Reader in front:

func sum(path string) {
	start := time.Now()
	r := csv.NewReader(bufio.NewReaderSize(open(path), 1<<20))
	r.ReuseRecord = true
	if _, err := r.Read(); err != nil {
		panic(err)
	}
	var total float64
	rows := 0
	for {
		rec, err := r.Read()
		if err == io.EOF {
			break
		}
		if err != nil {
			panic(err)
		}
		v, _ := strconv.ParseFloat(rec[3], 64)
		total += v
		rows++
	}
	fmt.Printf("rows=%d total=%.2f time=%.2fs peak_rss=%dMB\n", rows, total, time.Since(start).Seconds(), peakRSS())
}

cd go && go build -o /tmp/csvlab . && /tmp/csvlab sum ../data/orders-2m.csv && /tmp/csvlab readall ../data/orders-2m.csv

rows=2000000 total=160174646.36 time=0.98s peak_rss=7MB
ReadAll rows=2000000 total=160174646.36 time=1.60s peak_rss=496MB

The streaming loop summed 2,000,000 amounts to 160174646.36 in 0.98 s with a peak resident size of 7 MB. ReadAll, which returns every record as a slice of slices, took 1.60 s and 496 MB. The memory is the point: the stream's footprint does not depend on the file. Remember what ReuseRecord costs: the slice is overwritten by the next Read, so copy any field you keep (strings.Clone, or build your struct inside the loop).

For comparison, the Rust csv crate took 0.24 to 0.42 s on this file, Python's csv.reader 1.52 s, Java's OpenCSV 1.13 s and Node's csv-parse 4.16 s, with the machine and the work (summing one column) held constant. Go lands in the middle, and a single-threaded read is nearly always bounded by the parser and the disk.

Writing

csv.NewWriter writes to any io.Writer, quotes only when it must, and buffers:

func write() {
	w := csv.NewWriter(os.Stdout)
	w.UseCRLF = false
	w.Write([]string{"id", "name", "note"})
	w.Write([]string{"1", "Smith, Jane", "said \"hi\""})
	w.Write([]string{"2", "Zoë", "line one\nline two"})
	w.Write([]string{"3", " padded", ""})
	w.Flush()
	fmt.Println("w.Error():", w.Error())
}

cd go && go run . write

id,name,note
1,"Smith, Jane","said ""hi"""
2,Zoë,"line one
line two"
3," padded",
w.Error(): <nil>

Smith, Jane is quoted, the doubled quotes in said ""hi"" are right, and the line break forces quotes around the whole field. " padded" was quoted as well: the writer quotes a field with a leading space, which is a courtesy to readers that trim. The 3 record's empty note is just a trailing comma. Three habits: call Flush() before you exit or the last block is lost, check w.Error() after it, and set UseCRLF = true if the consumer expects Windows line endings (the default is \n). The writer converts nothing, so format numbers, dates and booleans yourself, with strconv and time.Format, and decide on null.

How to tell it worked

Count the records and compare with an independent count, remembering that a field with a line break makes more lines than records. Print the first header with %q so an invisible BOM shows. Check the error returned by Read on every call, not only at the end. Run the loop on a file with a short record and one with an unclosed quote, and make sure the exit status is non-zero. And for a file you write, read it back with another tool such as DuckDB and compare the counts.

What the package leaves to you

It does not convert types, strip a BOM, decode other encodings, name the columns, or tell you which of a set of ragged records are the real ones. It will not stream into a struct, so typed reading is a loop you write, or a third-party package. And its strictness about quotes is a feature that you meet at the worst time, with a file from a vendor whose quoting is almost right: LazyQuotes is the escape hatch, and the better fix is a conversation with the vendor. For a file that needs more than a loop, or for a quick question, a SQL query on the file is shorter than the Go.