Parsing CSV in Rust With the csv Crate and Serde Types

Rust's csv crate, by the author of ripgrep, is the CSV library that other languages' libraries get compared to: it is fast, it streams, it deserializes straight into structs through serde and its reader is configurable in every direction. It is also precise about the format in ways that matter, and it makes a design choice that surprised me, which is to treat an unclosed quote as valid. This article reads the series' test files with version 1.4.0, shows what the three record APIs cost on 2,000,000 rows, and covers the errors. The comparison with other languages is in the guide.

Setup

A project with two dependencies:

[package]
name = "csvlab"
version = "0.1.0"
edition = "2021"

[dependencies]
csv = "1"
serde = { version = "1", features = ["derive"] }

[profile.release]
opt-level = 3

csv is the reader and writer. serde with the derive feature lets you read into your own types.

Reading records

csv::Reader::from_path opens a file, treats the first row as a header and returns an iterator of Result<StringRecord, Error>:

fn people(path: &str) -> Result<(), Box<dyn Error>> {
    let mut rdr = csv::Reader::from_path(path)?;
    println!("headers: {:?}", rdr.headers()?);
    for rec in rdr.records() {
        let rec = rec?;
        let pos = rec.position().map(|p| p.line()).unwrap_or(0);
        println!("line {}: {:?}", pos, rec);
    }
    Ok(())
}

cargo run --release -q -- people ../data/people.csv

headers: StringRecord(["id", "name", "city", "note"])
line 2: StringRecord(["1", "Ana Silva", "Lisbon", "likes tea"])
line 3: StringRecord(["2", "Smith, Jane", "Boston", "said \"hello\""])
line 4: StringRecord(["3", "Bo Chen", "Oslo", "line one\nline two"])
line 6: StringRecord(["4", "Cy Dubois", "Lyon", ""])

Each record keeps its quoting out of the way: Smith, Jane is one field, said "hello" has its quotes resolved, and the note with a line break has \n in it. rec.position() returns where the record began, and .line() gave 2, 3, 4 and 6 (the third record spans two lines). The header row is separate: rdr.headers() returned ["id", "name", "city", "note"], and the record iterator starts at the data.

The crate's version, from Cargo.lock:

cargo build --release 2>&1 | tail -3 && grep -A1 'name = "csv"' Cargo.lock

   Compiling csv v1.4.0
   Compiling csvlab v0.1.0 (/b/lab/lang/rust)
    Finished `release` profile [optimized] target(s) in 3.74s
name = "csv"
version = "1.4.0"

Field counts

By default every record must have the same number of fields as the first, and a mismatch is an error carrying the position:

fn widths(path: &str, flexible: bool) -> Result<(), Box<dyn Error>> {
    let mut rdr = csv::ReaderBuilder::new().flexible(flexible).from_path(path)?;
    let mut n = 0;
    for rec in rdr.records() {
        match rec {
            Ok(_) => n += 1,
            Err(e) => {
                println!("error: {}", e);
                if let csv::ErrorKind::UnequalLengths { pos, expected_len, len } = e.kind() {
                    println!("  kind: UnequalLengths, line {:?}, expected {}, got {}", pos.as_ref().map(|p| p.line()), expected_len, len);
                }
            }
        }
    }
    println!("records read: {}", n);
    Ok(())
}

cargo run --release -q -- widths ../data/ragged2.csv

error: CSV error: record 2 (line: 3, byte: 41): found record with 5 fields, but the previous record has 4 fields
  kind: UnequalLengths, line Some(3), expected 4, got 5
error: CSV error: record 4 (line: 5, byte: 85): found record with 5 fields, but the previous record has 4 fields
  kind: UnequalLengths, line Some(5), expected 4, got 5
error: CSV error: record 5 (line: 6, byte: 101): found record with 3 fields, but the previous record has 4 fields
  kind: UnequalLengths, line Some(6), expected 4, got 3
error: CSV error: record 7 (line: 8, byte: 133): found record with 1 fields, but the previous record has 4 fields
  kind: UnequalLengths, line Some(8), expected 4, got 1
records read: 4

cargo run --release -q -- widths ../data/ragged2.csv flexible

records read: 8

The strict run printed four UnequalLengths errors, each with a record number, line, byte offset, the length found and the expected one (found record with 5 fields, but the previous record has 4 fields), and error.kind() gave a csv::ErrorKind::UnequalLengths { pos, expected_len, len } that you can match on. The loop continued after each error, and counted 4 good records of the 8. With flexible(true) all 8 came through, unchecked. Note the wording: the check compares each record with the previous one, not with the header. A file whose width changes once and stays changed fails once.

Into your own types

serde deserialization is where the crate shines. A struct whose field names match the header:

#[derive(Debug, Deserialize)]
struct Zip {
    id: u32,
    zip: String,
    state: String,
}

#[derive(Debug, Deserialize)]
struct ZipNum {
    id: u32,
    zip: u32,
    state: String,
}

fn serde_zips(path: &str) -> Result<(), Box<dyn Error>> {
    let mut rdr = csv::Reader::from_path(path)?;
    for rec in rdr.deserialize::<Zip>() {
        println!("as String: {:?}", rec?);
    }
    let mut rdr = csv::Reader::from_path(path)?;
    for rec in rdr.deserialize::<ZipNum>() {
        println!("as u32   : {:?}", rec?);
    }
    Ok(())
}

cargo run --release -q -- serde ../data/zips.csv

as String: Zip { id: 1, zip: "02134", state: "MA" }
as String: Zip { id: 2, zip: "90210", state: "CA" }
as String: Zip { id: 3, zip: "00501", state: "NY" }
as String: Zip { id: 4, zip: "10001", state: "NY" }
as u32   : ZipNum { id: 1, zip: 2134, state: "MA" }
as u32   : ZipNum { id: 2, zip: 90210, state: "CA" }
as u32   : ZipNum { id: 3, zip: 501, state: "NY" }
as u32   : ZipNum { id: 4, zip: 10001, state: "NY" }

With zip: String the values were "02134", "90210", "00501", "10001". With zip: u32 they were 2134, 90210, 501 and 10001, because the number parser accepts leading zeros and produces a number that has no memory of them. Which type is right depends on what the column is; for an identifier, String, and the compiler holds you to it everywhere the value is used. The Order struct in the same file deserializes only three of the eight columns, since extra columns are ignored.

A blank cell is a type error, not a default:

cargo run --release -q -- zipsblank ../data/zips-blank.csv

ok: ZipNum { id: 1, zip: 2134, state: "MA" }
error: CSV deserialize error: record 2 (line: 3, byte: 24): field 1: cannot parse integer from empty string
ok: ZipNum { id: 3, zip: 501, state: "NY" }

The first record parsed, the second failed with CSV deserialize error: record 2 (line: 3, byte: 24): field 1: cannot parse integer from empty string, and the third parsed again. This is the good version of the Java bean default: an empty u32 is an error you can see. If blanks are legal, make the field Option<u32>, which turns an empty cell into None.

The unclosed quote is not an error

Most parsers in this series report a quote that never closes. The csv crate does not:

cargo run --release -q -- people ../data/unterminated.csv | tail -n +2 | awk '{print "record", NR, "-", length($0), "bytes of text"}'
cargo run --release -q -- people ../data/unterminated.csv | wc -l

record 1 - 42 bytes of text
record 2 - 231623 bytes of text
3

The output shows two records. The first is a normal line. The second is 231,623 bytes of debug text: the field that began at the stray quote on line 3 and ran to the end of the file, swallowing about 5,000 lines. There was no error and the exit status was 0. RFC 4180 does not allow it, and the crate reads the invalid file leniently by closing the field at the end of input, which means you have to check for it yourself; how CSV quoting works explains the rule. Two cheap checks: a field longer than any value your data should hold, and a record count compared with qsv count. Go, Java, Ruby and Kotlin raise on the same file; Rust, C# and PHP do not.

BOM and encodings

The reader removed the byte order mark: for bom.csv the header came back as ["id", "name"], which is the friendliest behavior of the languages tested. A file in another encoding is on you. The reader works on bytes and a StringRecord insists on UTF-8, returning a Utf8 error for invalid bytes; ByteRecord skips that check and lets you decode as you like, with the encoding_rs crate for example.

Which record type

The crate gives three ways to read, with different costs. On the 2,000,000 row file, summing the amount column:

fn sum(path: &str, mode: &str) -> Result<(), Box<dyn Error>> {
    let start = Instant::now();
    let mut rdr = csv::ReaderBuilder::new().from_reader(File::open(path)?);
    let (mut total, mut rows) = (0.0f64, 0u64);
    match mode {
        "string" => {
            for rec in rdr.records() {
                total += rec?[3].parse::<f64>()?;
                rows += 1;
            }
        }
        "byte" => {
            let mut rec = csv::ByteRecord::new();
            while rdr.read_byte_record(&mut rec)? {
                total += std::str::from_utf8(&rec[3])?.parse::<f64>()?;
                rows += 1;
            }
        }
        "serde" => {
            for rec in rdr.deserialize::<Order>() {
                total += rec?.amount;
                rows += 1;
            }
        }
        _ => {}
    }
    println!("{:<7} rows={} total={:.2} time={:.2}s peak_rss={}MB", mode, rows, total, start.elapsed().as_secs_f64(), peak_rss_mb());
    Ok(())
}

for m in string byte serde; do cargo run --release -q -- sum ../data/orders-2m.csv $m; done

string  rows=2000000 total=160174646.36 time=0.42s peak_rss=1MB
byte    rows=2000000 total=160174646.36 time=0.24s peak_rss=1MB
serde   rows=2000000 total=160174646.36 time=0.40s peak_rss=1MB

• StringRecord (the records() iterator, allocating each record): 0.42 s.

• ByteRecord read into one reused buffer with read_byte_record: 0.24 s. No allocation per record and no UTF-8 validation of the fields you skip.

• deserialize::<Order>(): 0.40 s, with the types parsed for you. It is nearly as fast as StringRecord and much more convenient.

All three totaled 160174646.36 and showed a peak resident size of about 1 MB, since the reader streams through a small buffer. For comparison, Go's encoding/csv took 0.98 s on this file, Python's csv.reader 1.52 s, Java's OpenCSV 1.13 s and PHP's fgetcsv 5.49 s. A ByteRecord loop is the fastest I measured, and DuckDB's multi-threaded scan, at 0.2 s, is the same order for the whole job without writing any Rust.

Writing

csv::Writer quotes when it must and buffers:

fn write() -> Result<(), Box<dyn Error>> {
    let mut wtr = csv::Writer::from_writer(std::io::stdout());
    wtr.write_record(["id", "name", "note"])?;
    wtr.write_record(["1", "Smith, Jane", "said \"hi\""])?;
    wtr.write_record(["2", "Zoë", "line one\nline two"])?;
    wtr.write_record(["3", " padded", ""])?;
    wtr.flush()?;
    Ok(())
}

cargo run --release -q -- write

id,name,note
1,"Smith, Jane","said ""hi"""
2,Zoë,"line one
line two"
3, padded,

Smith, Jane and said ""hi"" are quoted correctly, the line break forces quotes, and " padded" is written without quotes: 3, padded,. That differs from Go, which quoted the leading space. Both are valid; a reader that trims spaces will treat them differently, so quote such fields yourself if it matters (QuoteStyle::Always or Necessary on the builder). Call flush(), since dropping the writer flushes but swallows errors.

How to tell it worked

Match the record count against an independent count. Check each error: iterate records() and handle every Err, not just the first. Print the length of your longest field, which catches the unclosed quote. For serde, read a file with a blank in every column you typed. Run cargo clippy, and keep the csv version in Cargo.lock.

What the crate does not tell you

It does not tell you that a quote never closed, it does not know the encoding of your file, and it does not decide what a blank means for you: serde makes you choose Option or fail. Its check of field counts is relative to the previous record. For a data pipeline on top of the crate, add the checks that the grammar permits and your data does not: maximum field length, the header you expect, the count of records. The format they all follow is RFC 4180, and the same task in the shell is in the shell guide.