Parsing CSV in Ruby With the CSV Library and Its Converters

Ruby's CSV library is one of the most complete in this series: it has header handling, converters, a lenient mode, a writer with options for every taste, and a clean error class. It is also the one that most invites you to turn on a convenience that changes your data. The :numeric converter is the example, and it does something I did not expect to a ZIP code. This article uses Ruby 3.4 and the csv gem, shows the change that broke a lot of Gemfiles in 2024, reads the series' test files, and times 2,000,000 rows. The comparison with other languages is in the guide.

csv is a bundled gem now

Until Ruby 3.3, require "csv" worked everywhere because csv was a default gem. Ruby 3.4 made it a bundled gem. A script run directly still finds it, but under Bundler the Gemfile has to name it:

mkdir -p /tmp/app && cd /tmp/app && printf 'source "https://rubygems.org"\n' > Gemfile && bundle install --quiet 2>&1 | tail -n 1
echo "--- without csv in the Gemfile:"
bundle exec ruby -e 'require "csv"; puts CSV::VERSION' 2>&1 | head -n 3
printf 'source "https://rubygems.org"\ngem "csv"\n' > Gemfile && bundle install --quiet 2>&1 | tail -n 1
echo "--- with gem \"csv\":"
bundle exec ruby -e 'require "csv"; puts CSV::VERSION' 2>&1 | head -n 3

The Gemfile specifies no dependencies
--- without csv in the Gemfile:
-e:1: warning: csv was loaded from the standard library, but is not part of the default gems starting from Ruby 3.4.0.
You can add csv to your Gemfile or gemspec to silence this warning.
/usr/local/lib/ruby/3.4.0/bundled_gems.rb:82:in 'Kernel.require': cannot load such file -- csv (LoadError)
--- with gem "csv":
3.3.6

With an empty Gemfile, bundle exec ruby -e 'require "csv"' printed warning: csv was loaded from the standard library, but is not part of the default gems starting from Ruby 3.4.0 and then cannot load such file -- csv (LoadError). With gem "csv" in the Gemfile it loaded, and the version was 3.3.6 from RubyGems, newer than the 3.3.2 that ships with the image. If a deploy that worked on Ruby 3.3 fails after the upgrade with a LoadError for csv, this is why, and the fix is one line in the Gemfile.

Reading with headers

CSV.foreach streams a file one row at a time, and headers: true turns each row into a CSV::Row that behaves like a hash:

require "csv"
path = ARGV[0]
puts "csv #{CSV::VERSION}, ruby #{RUBY_VERSION}"
CSV.foreach(path, headers: true) { |row| p row.to_h }

ruby people.rb ../data/people.csv

csv 3.3.2, ruby 3.4.12
{"id" => "1", "name" => "Ana Silva", "city" => "Lisbon", "note" => "likes tea"}
{"id" => "2", "name" => "Smith, Jane", "city" => "Boston", "note" => "said \"hello\""}
{"id" => "3", "name" => "Bo Chen", "city" => "Oslo", "note" => "line one\nline two"}
{"id" => "4", "name" => "Cy Dubois", "city" => "Lyon", "note" => nil}

The first line is the versions. The rows come back with string keys, Smith, Jane in one value, said "hello" with its quotes resolved and the line break in the third note. Look at the fourth record: "note" => nil. The note in the file is an empty field, and Ruby returns nil for an empty unquoted field. That is a distinction other libraries make you work for, and it comes with a twist, shown next.

nil and the empty string are different

An unquoted empty field is nil. A field written as "" is an empty string. CSV.parse_line on a line that has both:

ruby -rcsv -e 'p CSV.parse_line(%q(1,"Smith, Jane",,"")); p CSV.parse_line("a;b", col_sep: ";"); p CSV.parse_line(%q(a,"b"c,d)) rescue puts "#{$!.class}: #{$!.message}"'

["1", "Smith, Jane", nil, ""]
["a", "b"]
CSV::MalformedCSVError: Any value after quoted field isn't allowed in line 1.

1,"Smith, Jane",,"" parsed as ["1", "Smith, Jane", nil, ""]: the third field is nil and the fourth is "". The other results in the same output: a semicolon line with col_sep: ";" gave ["a", "b"], and a,"b"c,d raised CSV::MalformedCSVError: Any value after quoted field isn't allowed in line 1. The library is strict about a closing quote followed by text. The two empty cases are not always preserved by the writer's counterparts elsewhere, and you can fold them with options:

ruby -rcsv -e 'CSV.foreach("../data/people.csv", headers: true) { |r| p r["note"] } ; puts "with nil_value / empty_value:"; CSV.foreach("../data/people.csv", headers: true, nil_value: "", empty_value: nil) { |r| p r["note"] }'

"likes tea"
"said \"hello\""
"line one\nline two"
nil
with nil_value / empty_value:
"likes tea"
"said \"hello\""
"line one\nline two"
""

By default the last note was nil. With nil_value: "" and empty_value: nil, the roles swapped and the note became "". Choose one meaning for "no value" in your application, and set both options so that it is the only one you see. Code that calls .strip on a field raises NoMethodError on the nil, which is how most people find out.

The numeric converter and the octal trap

Converters change values as they are read. :numeric converts anything that looks like an integer or a float, and :all adds dates. Here is the ZIP file with the first record shown under five configurations:

require "csv"
CSV.foreach(ARGV[0], headers: true) { |row| puts "default      : #{row.to_h}" ; break }
CSV.foreach(ARGV[0], headers: true, converters: :numeric) { |row| puts "numeric      : #{row.to_h}"; break }
CSV.foreach(ARGV[0], headers: true, converters: :all) { |row| puts "all          : #{row.to_h}"; break }
zip_as_text = ->(field, info) { info.header == "zip" ? field : (Integer(field, exception: false) || field) }
CSV.foreach(ARGV[0], headers: true, converters: zip_as_text) { |row| puts "custom       : #{row.to_h}"; break }
CSV.foreach(ARGV[0], headers: true, header_converters: :symbol) { |row| puts "symbol keys  : #{row.to_h}"; break }

ruby converters.rb ../data/zips.csv

default      : {"id" => "1", "zip" => "02134", "state" => "MA"}
numeric      : {"id" => 1, "zip" => 1116, "state" => "MA"}
all          : {"id" => 1, "zip" => 1116, "state" => "MA"}
custom       : {"id" => 1, "zip" => "02134", "state" => "MA"}
symbol keys  : {id: "1", zip: "02134", state: "MA"}

The default left everything a string: "zip" => "02134". converters: :numeric returned "zip" => 1116. That is not a typo. Ruby's integer parsing treats a leading zero as octal, so 02134 is the octal number 2134, which is 1116 in decimal (2×512 + 1×64 + 3×8 + 4). The ZIP code was not just stripped of its zero, as in pandas or C#; it was converted to a different number. :all did the same. The damage depends on the digits, which I checked on four ZIP codes:

require "csv"
data = "zip\n02134\n00501\n08902\n90210\n"
CSV.parse(data, headers: true, converters: :numeric).each { |r| puts "#{r['zip'].inspect} (#{r['zip'].class})" }
puts "Integer(\"02134\", 10) = #{Integer('02134', 10)}; Integer(\"0x1F\") = #{Integer('0x1F')}; Integer(\"08902\") raises: #{(Integer('08902') rescue $!.class)}"

ruby octal.rb

1116 (Integer)
321 (Integer)
8902.0 (Float)
90210 (Integer)
Integer("02134", 10) = 2134; Integer("0x1F") = 31; Integer("08902") raises: ArgumentError

02134 became the Integer 1116, 00501 became the Integer 321, 90210 stayed 90210, and 08902 became the Float 8902.0, because 8 is not an octal digit, Integer("08902") raises ArgumentError, and the converter falls back to a float. So one column ends up as a mixture of wrong integers, right integers and floats, which is worse than a uniform loss.

The fix is a converter that knows about columns. A lambda receives the field and an info object with the header:

zip_as_text = ->(field, info) { info.header == "zip" ? field : (Integer(field, exception: false) || field) }

The custom line of the output shows it: "id" => 1, "zip" => "02134". Note that even this uses Integer(field), which accepts a 0x prefix (Integer("0x1F") is 31) and a leading zero as octal. Integer(field, 10) forces base 10 (Integer("02134", 10) is 2134), and is what I would use for the columns that are numbers. And header_converters: :symbol turned the keys into {id: "1", zip: "02134", ...}, which is convenient and changes the key type, so a lookup by string stops working.

Field counts, bad quotes and the BOM

Ruby does not check how many fields each row has:

require "csv"
[["ragged2.csv", {}], ["ragged2.csv", {liberal_parsing: true}], ["unterminated.csv", {}], ["ragged2.csv", {headers: true}]].each do |file, opts|
  n = 0
  begin
    CSV.foreach("../data/#{file}", **opts) { n += 1 }
    puts "#{file} #{opts}: #{n} rows, no error"
  rescue CSV::MalformedCSVError => e
    puts "#{file} #{opts}: #{e.class}: #{e.message} (line #{e.line_number})"
  end
end
puts "--- field count is not checked:"
rows = CSV.read("../data/ragged2.csv")
p rows.map(&:size)
puts "--- with headers, a short row has nil, a long one has a nil header:"
t = CSV.read("../data/ragged2.csv", headers: true)
p t[1].to_h, t[2].to_h

ruby errors.rb

ragged2.csv {}: 9 rows, no error
ragged2.csv {liberal_parsing: true}: 9 rows, no error
unterminated.csv {}: CSV::MalformedCSVError: Unclosed quoted field in line 3. (line 3)
ragged2.csv {headers: true}: 8 rows, no error
--- field count is not checked:
[4, 4, 5, 4, 5, 3, 4, 1, 4]
--- with headers, a short row has nil, a long one has a nil header:
{"id" => "2", "name" => "Bo", "city" => "Oslo", "note" => "likes coffee", nil => " no sugar"}
{"id" => "3", "name" => "Cy", "city" => "Lyon", "note" => nil}

The ragged file read as 9 rows (8 with headers: true), with widths [4, 4, 5, 4, 5, 3, 4, 1, 4] and no error, and liberal_parsing: true changed nothing on it. With headers, the row with an extra field has a nil key holding the extra value (nil => " no sugar"), and a short row has nil for the missing fields. The unclosed quote is reported: CSV::MalformedCSVError: Unclosed quoted field in line 3., with line_number 3, the line where the quote opened, which is the useful one.

The BOM was dropped on its own:

ruby bom.rb

["id", "name"]
["id", "name"]

Both the plain read and the one with encoding: "bom|utf-8" gave ["id", "name"]. I would still pass the encoding when you control it, since the explicit form is the documented one and works on older csv versions.

Writing

CSV.generate builds a string, CSV.open writes a file, both with <<:

require "csv"
puts CSV.generate { |csv|
  csv << %w[id name note]
  csv << [1, "Smith, Jane", 'said "hi"']
  csv << [2, "Zoë", "line one\nline two"]
  csv << [3, " padded", nil]
  csv << [4, "=SUM(A1)", 0.1 + 0.2]
  csv << [5, true, ""]
}
puts "--- force_quotes and a semicolon:"
puts CSV.generate(force_quotes: true, col_sep: ";") { |csv| csv << [1, "a;b", nil, ""] }

ruby write.rb

id,name,note
1,"Smith, Jane","said ""hi"""
2,Zoë,"line one
line two"
3, padded,
4,=SUM(A1),0.30000000000000004
5,true,""
--- force_quotes and a semicolon:
"1";"a;b";"";""

Smith, Jane and said ""hi"" are quoted; the line break forces quotes; " padded" is not quoted. nil is written as an empty field, and an empty string "" is written as "", so the nil/empty distinction survives a round trip through Ruby. 0.1 + 0.2 was written as 0.30000000000000004, true as true, and =SUM(A1) unguarded. force_quotes: true with col_sep: ";" produced "1";"a;b";"";"", with the nil also quoted as "".

Speed on 2,000,000 rows

Three ways to sum the amount column:

require "csv"
mode = ARGV[1]
t = Process.clock_gettime(Process::CLOCK_MONOTONIC)
total = 0.0; rows = 0
case mode
when "foreach"
  CSV.foreach(ARGV[0], headers: true) { |r| total += r["amount"].to_f; rows += 1 }
when "arrays"
  first = true
  CSV.foreach(ARGV[0]) { |r| (first = false; next) if first; total += r[3].to_f; rows += 1 }
when "read"
  CSV.read(ARGV[0], headers: true).each { |r| total += r["amount"].to_f; rows += 1 }
end
rss = File.read("/proc/self/status")[/VmHWM:\s+(\d+)/, 1].to_i / 1024
printf("%-8s rows=%d total=%.2f time=%.2fs peak_rss=%dMB\n", mode, rows, total, Process.clock_gettime(Process::CLOCK_MONOTONIC) - t, rss)

for m in arrays foreach read; do ruby sum.rb ../data/orders-2m.csv $m; done

arrays   rows=2000000 total=160174646.36 time=9.66s peak_rss=14MB
foreach  rows=2000000 total=160174646.36 time=14.95s peak_rss=14MB
read     rows=2000000 total=160174646.36 time=25.72s peak_rss=1974MB

Reading rows as arrays took 9.66 s at 14 MB. Using headers: true to get hash-like rows took 14.95 s, so the header convenience costs about 55% here. CSV.read(...), which loads the whole table, took 25.72 s and reached 1,974 MB. All three totaled 160174646.36. Ruby is the slowest of the languages tested, about six times Python's csv.reader on arrays (1.52 s) and forty times Rust's ByteRecord loop (0.24 s), and the difference is the interpreter running a block for each row. If the file is big, stream with foreach, skip the headers if you can, or let DuckDB or COPY do the work. For a quick job on a small file, none of this matters.

How to tell it worked

Count rows against another tool. Print the class of a few values, to see nil and Integer where you expect them. Read a ZIP column and a column of numbers with leading zeros. Run a file with a short row, a long row and an unclosed quote. Under Bundler, run bundle exec ruby -e 'require "csv"' on a clean checkout. And write the output back and read it with a different tool.

What the converters and defaults do to data

:numeric and :all are tempting on a quick script and wrong for identifiers, with octal on top. nil for empty is helpful and surprising. No field-count check, no BOM check needed, and a strict error for the quotes. The safe habit is to read everything as strings, convert in one function that you wrote, and keep the identifier columns out of it. For a large import, COPY in PostgreSQL or a DuckDB query removes the loop; the format reference is RFC 4180.