Reading a CSV File in Kotlin for a Quick Import Job
Read a CSV in Kotlin with kotlin-csv and build a small import job that validates rows, rejects bad ones with row numbers and never hides a failure.
A Kotlin service often needs a CSV import: a nightly file from a partner, an admin upload, a seed script. The Kotlin standard library has no CSV parser, so the choices are a Java library (OpenCSV, Commons CSV) or a Kotlin one. This article uses kotlin-csv (the doyaaaaaken library), shows how to turn rows into a data class with validation and a list of rejects, and checks what the library does with the awkward files of this series. The comparison with other languages is in the guide, and the Java route is in the OpenCSV article.
Why not split
The one-line version is File(path).readLines().map { it.split(",") }, and it fails on the first quoted comma:
"naive" -> File(args[1]).readLines().forEach { println(it.split(",")) }chmod +x /opt/kt/kotlinc/bin/*; export PATH=/opt/kt/kotlinc/bin:$PATH; kotlinc -version 2>&1 | tail -n 1; kotlinc Lab.kt -cp /opt/kt/kotlin-csv.jar -d /tmp/lab 2>&1 | grep -v warning | head -n 5; java -cp /tmp/lab:/opt/kt/kotlin-csv.jar:/opt/kt/kotlinc/lib/kotlin-stdlib.jar LabKt version; unzip -p /opt/kt/kotlin-csv.jar META-INF/MANIFEST.MF 2>/dev/null | head -n 0; echo "kotlin-csv-jvm 1.10.0"info: kotlinc-jvm 2.4.21 (JRE 21.0.12.1+1-LTS)
Kotlin 2.4.21, JVM 21.0.12.1
kotlin-csv-jvm 1.10.0chmod +x /opt/kt/kotlinc/bin/*; export PATH=/opt/kt/kotlinc/bin:$PATH; kotlinc Lab.kt -cp /opt/kt/kotlin-csv.jar -d /tmp/lab 2>&1 | grep -v warning | head -n 5; java -cp /tmp/lab:/opt/kt/kotlin-csv.jar:/opt/kt/kotlinc/lib/kotlin-stdlib.jar LabKt naive ../data/people.csv[id, name, city, note]
[1, Ana Silva, Lisbon, likes tea]
[2, "Smith, Jane", Boston, "said ""hello"""]
[3, Bo Chen, Oslo, "line one]
[line two"]
[4, Cy Dubois, Lyon, ]"Smith, Jane" is two list elements with the quotes still attached, and the note with a line break is cut into [3, Bo Chen, Oslo, "line one] and [line two"], a second record with one field. This is not a Kotlin problem, it is the delimiter-splitting problem as in every language. Use a parser.
Reading rows as maps
kotlin-csv's reader is configured with a DSL and offers readAll, readAllWithHeader and sequence versions:
"people" -> {
val rows = csvReader().readAllWithHeader(File(args[1]))
rows.forEach { println(it.mapValues { (_, v) -> v.replace("\n", "\\n") }) }
}chmod +x /opt/kt/kotlinc/bin/*; export PATH=/opt/kt/kotlinc/bin:$PATH; kotlinc Lab.kt -cp /opt/kt/kotlin-csv.jar -d /tmp/lab 2>&1 | grep -v warning | head -n 5; java -cp /tmp/lab:/opt/kt/kotlin-csv.jar:/opt/kt/kotlinc/lib/kotlin-stdlib.jar LabKt people ../data/people.csv{id=1, name=Ana Silva, city=Lisbon, note=likes tea}
{id=2, name=Smith, Jane, city=Boston, note=said "hello"}
{id=3, name=Bo Chen, city=Oslo, note=line one\nline two}
{id=4, name=Cy Dubois, city=Lyon, note=}Each row is a Map<String, String> keyed by the header. Smith, Jane is one value, said "hello" has its quotes resolved, and the note has its line break (printed as \n). An empty cell is the empty string "", not null. Kotlin's type system helps from here: map the strings to a data class and make the nullable fields nullable.
"typed" -> {
val people = csvReader().readAllWithHeader(File(args[1])).map {
Person(it.getValue("id").toInt(), it.getValue("name"), it.getValue("city"), it["note"]?.ifEmpty { null })
}
people.forEach { println(it.copy(note = it.note?.replace("\n", "\\n"))) }
}chmod +x /opt/kt/kotlinc/bin/*; export PATH=/opt/kt/kotlinc/bin:$PATH; kotlinc Lab.kt -cp /opt/kt/kotlin-csv.jar -d /tmp/lab 2>&1 | grep -v warning | head -n 5; java -cp /tmp/lab:/opt/kt/kotlin-csv.jar:/opt/kt/kotlinc/lib/kotlin-stdlib.jar LabKt typed ../data/people.csvPerson(id=1, name=Ana Silva, city=Lisbon, note=likes tea)
Person(id=2, name=Smith, Jane, city=Boston, note=said "hello")
Person(id=3, name=Bo Chen, city=Oslo, note=line one\nline two)
Person(id=4, name=Cy Dubois, city=Lyon, note=null)Person(id=4, ..., note=null): the ifEmpty { null } turned the blank note into a real null, and the first three records kept their text. Note the order in the constructor call: getValue("id") throws NoSuchElementException if a header is missing, which is the behavior I want from a column the code depends on.
Numbers and ZIP codes
String.toInt() parses a leading zero without ceremony, so a ZIP code stored as an Int loses its zero:
"zips" -> {
csvReader().readAllWithHeader(File(args[1])).forEach {
val z = it.getValue("zip")
println("text=$z toInt()=${z.toInt()} toIntOrNull()=${z.toIntOrNull()}")
}
}
"blank" -> {
csvReader().readAllWithHeader(File(args[1])).forEach {
val z = it.getValue("zip")
println("id=${it["id"]} zip='$z' toIntOrNull()=${z.toIntOrNull()}")
}
}chmod +x /opt/kt/kotlinc/bin/*; export PATH=/opt/kt/kotlinc/bin:$PATH; kotlinc Lab.kt -cp /opt/kt/kotlin-csv.jar -d /tmp/lab 2>&1 | grep -v warning | head -n 5; java -cp /tmp/lab:/opt/kt/kotlin-csv.jar:/opt/kt/kotlinc/lib/kotlin-stdlib.jar LabKt zips ../data/zips.csv; java -cp /tmp/lab:/opt/kt/kotlin-csv.jar:/opt/kt/kotlinc/lib/kotlin-stdlib.jar LabKt blank ../data/zips-blank.csvtext=02134 toInt()=2134 toIntOrNull()=2134
text=90210 toInt()=90210 toIntOrNull()=90210
text=00501 toInt()=501 toIntOrNull()=501
text=10001 toInt()=10001 toIntOrNull()=10001
id=1 zip='02134' toIntOrNull()=2134
id=2 zip='' toIntOrNull()=null
id=3 zip='00501' toIntOrNull()=501text=02134 toInt()=2134 toIntOrNull()=2134, and 00501 became 501. The second part of the output reads a file with a blank ZIP: toIntOrNull() returned null for the empty string, which is why I prefer it to toInt(), which would throw NumberFormatException. Keep identifiers as String in the data class. Use BigDecimal for money, not Double; the import job below does.
A small import job
The pattern for a job that must not fail silently is to turn each row into a Result, keep the good ones, and collect the bad ones with a row number:
data class Order(val id: Int, val customer: String, val qty: Int?, val price: java.math.BigDecimal, val status: String)
val STATUSES = setOf("paid", "refunded", "pending", "failed")
// Turn one row into an Order, or say why it cannot be one.
fun toOrder(row: Map<String, String>): Result<Order> = runCatching {
val qty = row.getValue("qty").takeIf { it.isNotBlank() }?.let { it.toIntOrNull() ?: error("qty '$it' is not a whole number") }
val price = java.math.BigDecimal(row.getValue("price"))
require(price.signum() >= 0) { "price $price is negative" }
val status = row.getValue("status")
require(status in STATUSES) { "unknown status '$status'" }
Order(row.getValue("id").toInt(), row.getValue("customer"), qty, price, status) "import" -> {
val good = mutableListOf<Order>()
val rejects = mutableListOf<String>()
var rowNo = 1 // the header is row 1
csvReader().open(File(args[1])) {
readAllWithHeaderAsSequence().forEach { row ->
rowNo++
toOrder(row).fold({ good += it }, { rejects += "row $rowNo: ${it.message}" })
}
}
println("imported ${good.size}, rejected ${rejects.size}")
rejects.forEach { println(" $it") }
println("total value of imported rows: ${good.sumOf { it.price * (it.qty ?: 1).toBigDecimal() }}")
}The first block defines an Order and toOrder, which wraps the conversion in runCatching: qty may be blank (null) but must be a whole number if present, price must be a non-negative BigDecimal, and status must be one of four values. The second reads the file lazily with readAllWithHeaderAsSequence(), calls toOrder, and uses fold to add either to good or, with the message, to rejects. Run on a five row file with three kinds of bad row:
chmod +x /opt/kt/kotlinc/bin/*; export PATH=/opt/kt/kotlinc/bin:$PATH; kotlinc Lab.kt -cp /opt/kt/kotlin-csv.jar -d /tmp/lab 2>&1 | grep -v warning | head -n 8; java -cp /tmp/lab:/opt/kt/kotlin-csv.jar:/opt/kt/kotlinc/lib/kotlin-stdlib.jar LabKt import ../data/orders-dirty.csvimported 2, rejected 3
row 3: qty 'x' is not a whole number
row 4: price -100.00 is negative
row 6: unknown status 'unknown'
total value of imported rows: 44.25imported 2, rejected 3, and three lines that say which row and why: row 3: qty 'x' is not a whole number, row 4: price -100.00 is negative, row 6: unknown status 'unknown'. The row numbers count records with the header as row 1, so row 3 is the second data record, the "Smith, Jane" order. The total value of the imported rows came to 44.25: 3 × 9.50 plus 15.75, with the blank quantity counted as one. That last choice, a blank quantity meaning one, is business logic that belongs in a comment and a test.
Three properties make the job safe to operate. It processes a sequence, so memory does not grow with the file. Every rejection has a reason and a row number, so a person can fix the source. And the summary line, imported 2, rejected 3, can be compared with expectations or turned into a non-zero exit code when the reject count passes a threshold. What it does not do is roll back: if you write to a database inside the forEach, wrap the whole import in a transaction or write the accepted rows in batches.
Ragged rows
kotlin-csv checks field counts, which several libraries here do not:
"ragged" -> {
try {
val rows = csvReader().readAllWithHeader(File(args[1]))
println("rows: ${rows.size}")
} catch (e: CSVFieldNumDifferentException) {
println("CSVFieldNumDifferentException: ${e.message}")
}
val lenient = csvReader {
insufficientFieldsRowBehaviour = com.github.doyaaaaaken.kotlincsv.dsl.context.InsufficientFieldsRowBehaviour.EMPTY_STRING
excessFieldsRowBehaviour = com.github.doyaaaaaken.kotlincsv.dsl.context.ExcessFieldsRowBehaviour.TRIM
}
val rows = lenient.readAllWithHeader(File(args[1]))
println("lenient rows: ${rows.size}; widths: ${rows.map { it.size }}")
}chmod +x /opt/kt/kotlinc/bin/*; export PATH=/opt/kt/kotlinc/bin:$PATH; kotlinc Lab.kt -cp /opt/kt/kotlin-csv.jar -d /tmp/lab 2>&1 | grep -v warning | head -n 5; java -cp /tmp/lab:/opt/kt/kotlin-csv.jar:/opt/kt/kotlinc/lib/kotlin-stdlib.jar LabKt ragged ../data/ragged2.csvCSVFieldNumDifferentException: Fields num seems to be 4 on each row, but on 2th csv row, fields num is 5.
lenient rows: 8; widths: [4, 4, 4, 4, 4, 4, 4, 4]By default the ragged file threw CSVFieldNumDifferentException: Fields num seems to be 4 on each row, but on 2th csv row, fields num is 5. That is a message with a row, and a fatal one, which is the right default. The lenient configuration, with insufficientFieldsRowBehaviour = EMPTY_STRING and excessFieldsRowBehaviour = TRIM, returned 8 rows, each with 4 values. The extra field was trimmed and the missing ones filled with empty strings, silently. That is the setting to use only for a feed you know has trailing optional fields, and with a count of how many rows it reshaped, which the library does not provide.
Bad quotes and the BOM
An unclosed quote throws, and the exception carries the lost text:
"unterminated" -> {
try {
val rows = csvReader().readAll(File(args[1]))
println("rows: ${rows.size}")
} catch (e: MalformedCSVException) {
println("MalformedCSVException; first line of the message: ${e.message?.lineSequence()?.first()}")
println("the message is ${e.message?.length} characters long")
}
}chmod +x /opt/kt/kotlinc/bin/*; export PATH=/opt/kt/kotlinc/bin:$PATH; kotlinc Lab.kt -cp /opt/kt/kotlin-csv.jar -d /tmp/lab 2>&1 | grep -v warning | head -n 5; java -cp /tmp/lab:/opt/kt/kotlin-csv.jar:/opt/kt/kotlinc/lib/kotlin-stdlib.jar LabKt unterminated ../data/unterminated.csvMalformedCSVException; first line of the message: "2,Bo,"oops this quote never closes
the message is 226650 characters longMalformedCSVException, and the message begins with the line where the quote opened, "2,Bo,"oops this quote never closes. The message is 226,650 characters long, because it includes the entire remaining text. Log only the first line, as the code above does, or a bad file will put a quarter of a megabyte in your logs. The BOM is handled:
chmod +x /opt/kt/kotlinc/bin/*; export PATH=/opt/kt/kotlinc/bin:$PATH; kotlinc Lab.kt -cp /opt/kt/kotlin-csv.jar -d /tmp/lab 2>&1 | grep -v warning | head -n 5; java -cp /tmp/lab:/opt/kt/kotlin-csv.jar:/opt/kt/kotlinc/lib/kotlin-stdlib.jar LabKt bom ../data/bom.csv["id", "name"]The header keys were "id" and "name" without a stray U+FEFF. That matches R, Rust and C#, and differs from Go, Java and PHP.
Speed
Three ways to sum the amount column of the 2,000,000 row file:
"sum" -> {
val t = System.nanoTime()
var total = 0.0; var rows = 0L
csvReader().open(File(args[1])) {
readAllAsSequence().drop(1).forEach { total += it[3].toDouble(); rows++ }
}
println("kotlin-csv rows=$rows total=${"%.2f".format(total)} time=${"%.2f".format((System.nanoTime() - t) / 1e9)}s peak_rss=${peakRssMb()}MB")
}
"sum-header" -> {
val t = System.nanoTime()
var total = 0.0; var rows = 0L
csvReader().open(File(args[1])) {
readAllWithHeaderAsSequence().forEach { total += it.getValue("amount").toDouble(); rows++ }
}
println("with header map rows=$rows total=${"%.2f".format(total)} time=${"%.2f".format((System.nanoTime() - t) / 1e9)}s peak_rss=${peakRssMb()}MB")
}
"sum-readall" -> {
val t = System.nanoTime()
val all = csvReader().readAll(File(args[1]))
var total = 0.0
for (r in all.drop(1)) total += r[3].toDouble()
println("readAll (list) rows=${all.size - 1} total=${"%.2f".format(total)} time=${"%.2f".format((System.nanoTime() - t) / 1e9)}s peak_rss=${peakRssMb()}MB")
}chmod +x /opt/kt/kotlinc/bin/*; export PATH=/opt/kt/kotlinc/bin:$PATH; kotlinc Lab.kt -cp /opt/kt/kotlin-csv.jar -d /tmp/lab 2>&1 | grep -v warning | head -n 5; for m in sum sum-header sum-readall; do java -cp /tmp/lab:/opt/kt/kotlin-csv.jar:/opt/kt/kotlinc/lib/kotlin-stdlib.jar LabKt $m ../data/orders-2m.csv; donekotlin-csv rows=2000000 total=160174646.36 time=3.89s peak_rss=173MB
with header map rows=2000000 total=160174646.36 time=4.92s peak_rss=175MB
readAll (list) rows=2000000 total=160174646.36 time=5.27s peak_rss=1352MBStreaming rows as lists took 3.89 s at 173 MB. Streaming with the header map, which builds a Map per row, took 4.92 s. readAll into a list took 5.27 s and 1,352 MB. All three totaled 160174646.36. The JVM figures include a heap that grows to what the JVM allows, so the point is the ratio: the stream stayed small and the list did not. Against the other languages, Kotlin with this library is slower than Java with OpenCSV (1.13 s) and than Go (0.98 s); the convenience of the maps is paid for in allocation. For a nightly job of a few hundred thousand rows, that does not matter.
How to tell it worked
Compare good.size + rejects.size with the number of records in the file, counted by a different tool. Read the reject messages; they should name the real problem. Run the job on a file with a blank quantity, a negative price, a short row, an unclosed quote and a BOM. Check the exit status when the reject count is above your threshold. And keep the reject list as a file that the owner of the source can use.
What the job does not guarantee
It does not make the import atomic, it does not know the encoding of the file (the JVM default is UTF-8 on modern JDKs, but pass a charset explicitly), and its rules, a non-negative price and four statuses, are only as good as the people who wrote them. It reads one file on one thread. For a very large load, hand the file to the database's bulk loader and validate in SQL with a staging table, as in the PostgreSQL article. The format reference is RFC 4180.
