CSVW and Describing CSV With Metadata That Tools Can Read

A CSV cannot describe itself. It does not say which character separates the fields, whether the first line is a header, which encoding it uses, whether 04.03.2026 is March or April, or whether the column called Betrag holds a decimal number or a code. Every reader makes guesses, and each guess is a place where two systems disagree. The usual fix is a README, which a person reads and a program cannot. CSVW, short for CSV on the Web, is the W3C's answer: a JSON file next to the CSV that states those facts in a vocabulary tools can parse. I have found it more useful as a precise way to write down a feed's contract than as something every library understands, and this article shows both sides: a real metadata file, a script that uses it to read a hard file correctly, and an honest account of where tool support runs out. The other half of the picture, the dialects the metadata describes, is in CSV dialects and variants you will run into.

What the standard gives you

CSVW is a set of W3C Recommendations from 2015, in two parts you will actually use. The data model says what a table is: columns, rows, cells and the datatypes of values. The metadata vocabulary says how to write that down as JSON. A metadata file is associated with a CSV by name: for orders.csv, the default is orders.csv-metadata.json in the same place, and a file called csv-metadata.json in the directory applies to every CSV there. A publisher can also link to the metadata through an HTTP header, so the CSV does not move.

Three groups of properties do most of the work. The dialect says how to read the text: delimiter, quoteChar, doubleQuote, header, headerRowCount, skipRows, commentPrefix, lineTerminators, encoding and trim. The table schema lists columns, each with a name, human titles (the header text in the file), a datatype and flags such as required. And the schema can declare a primaryKey and foreignKeys between tables, which is the part a plain data dictionary rarely states precisely.

A file that needs describing

Here is a file from a German accounting export. A comment line sits above the header, the delimiter is a semicolon, the dates are day-first with dots, and the amounts use a point for thousands and a comma for decimals:

Quarterly export, do not edit
Bestellung;Kunde;Datum;Betrag
1001;Müller GmbH;04.03.2026;1.234,50
1002;Dupont & Fils;05.03.2026;89,90

A sniffer will manage the delimiter and might skip the comment line. It will not know that 04.03.2026 is the fourth of March, and it will read 1.234,50 as text. The metadata states all of it:

{
  "@context": "http://www.w3.org/ns/csvw",
  "url": "eu-orders.csv",
  "dialect": {
    "delimiter": ";",
    "header": true,
    "skipRows": 1,
    "encoding": "utf-8"
  },
  "tableSchema": {
    "columns": [
      {"name": "order_id", "titles": "Bestellung", "datatype": "integer", "required": true},
      {"name": "customer", "titles": "Kunde", "datatype": "string"},
      {"name": "placed_on", "titles": "Datum", "datatype": {"base": "date", "format": "dd.MM.yyyy"}},
      {"name": "total", "titles": "Betrag", "datatype": {"base": "decimal", "format": {"decimalChar": ",", "groupChar": "."}}}
    ],
    "primaryKey": "order_id"
  }
}

Read it as a contract. The url names the file. The dialect skips one line before the header and splits on semicolons. Each column has a stable machine name (order_id) separate from the title in the file (Bestellung), which is useful when a provider renames a header and your schema should not follow. The datatype for the date gives its format, and for the amount gives the decimal and grouping characters. The primary key says which column identifies a row, so a validator can check for duplicates (finding duplicate keys in a CSV).

Using it to read the file

Most CSV readers do not read CSVW, so I wrote the smallest script that reads the properties above and builds a DuckDB query. It reads every column as text, then converts each using the datatype:

import json, sys
import duckdb

meta = json.load(open(sys.argv[1]))
d = meta["dialect"]
cols = meta["tableSchema"]["columns"]

TYPES = {"integer": "INTEGER", "string": "VARCHAR", "date": "VARCHAR", "decimal": "VARCHAR"}
columns = {c["name"]: TYPES[c["datatype"] if isinstance(c["datatype"], str) else c["datatype"]["base"]] for c in cols}

selects = []
for c in cols:
    dt = c["datatype"]
    name = c["name"]
    if isinstance(dt, dict) and dt["base"] == "date":
        fmt = dt["format"].replace("dd", "%d").replace("MM", "%m").replace("yyyy", "%Y")
        selects.append(f"strptime({name}, '{fmt}')::DATE AS {name}")
    elif isinstance(dt, dict) and dt["base"] == "decimal":
        g, dec = dt["format"]["groupChar"], dt["format"]["decimalChar"]
        selects.append(f"replace(replace({name}, '{g}', ''), '{dec}', '.')::DECIMAL(12,2) AS {name}")
    else:
        selects.append(name)

col_defs = ", ".join(f"'{k}': '{v}'" for k, v in columns.items())
sql = f"""
SELECT {', '.join(selects)}
FROM read_csv('{meta['url']}', delim='{d['delimiter']}', skip={d.get('skipRows', 0) + 1},
              header=false, columns={{{col_defs}}})
"""
for row in duckdb.sql(sql).fetchall():
    print(row)
print(duckdb.sql(sql).types)

(1001, 'Müller GmbH', datetime.date(2026, 3, 4), Decimal('1234.50'))
(1002, 'Dupont & Fils', datetime.date(2026, 3, 5), Decimal('89.90'))
[INTEGER, VARCHAR, DATE, DECIMAL(12,2)]

The result is what a person would want: an integer key, text, a real date (4 March, not 3 April) and an exact decimal of 1234.50. Nothing in the script knew the file was German. It followed the metadata, and the same script reads a different feed given a different JSON file. That is the practical argument for CSVW: the knowledge lives in a document, not in a reader's configuration scattered across scripts. The skip count adds one to skipRows because I read with header=false and name the columns from the schema, so the header line also has to be skipped; a fuller implementation would check that the header matches the titles.

Where support runs out

CSVW has been a Recommendation for years and adoption is uneven. The well-known conformance tooling lives in the Ruby and JavaScript ecosystems, some open-data portals publish CSVW metadata and validators exist, but the everyday readers, pandas, COPY, Excel and most loaders, ignore it. So three realities apply.

You will write glue, as above, or use a library for your language if one exists. The glue can be small, since the properties you need are a handful, and it gives you one place to put the dialect and the types.

The metadata does not stop a producer from writing a file that disagrees with it, and a file that ignores its own metadata is a more confusing failure than one with no metadata. Validate: a check that the header matches the declared titles, every value parses with its datatype and the primary key is unique should run on every file (validating a CSV file).

Excel and database tools will not read the file for you, so the metadata is documentation for them, which is still worth having. A sidecar JSON that says "semicolon, day-first dates, decimal comma" saves a long email.

CSVW or something simpler

The alternatives are lighter and also work. The Frictionless Data standard (Table Schema and Data Package) covers columns, types and constraints in a similar JSON form, has Python and JavaScript libraries and is better supported in data tooling than CSVW (writing a schema for a CSV). A SQL CREATE TABLE with comments is an unambiguous schema that your database already enforces. A shared README with a table of columns is enough for a feed with two consumers.

I would pick CSVW when the audience is public or crosses organizations and the standard itself matters (open data), or when the dialect is awkward and you want it stated in a place a program can read. For an internal feed, a schema in the language your loaders use is usually quicker to write and easier to enforce.

When CSVW is more than you need

If every reader of your CSV is code you control, and the dialect is the ordinary one, the metadata adds a file to maintain and nothing to the result. Put the settings in the loader, test them and move on. If the file is a one-off, a sentence in the email is cheaper than a vocabulary.

The case where a metadata file earns its place is the feed that outlives its authors: when the person who wrote the export has left, the file's metadata is what tells the next engineer that Datum is day-first and Betrag uses a decimal comma. Write it when you write the feed, keep it in the same repository as the sample files, and make the loader fail if the two disagree. The primer at the W3C (CSV on the Web primer) walks through examples, and the normative vocabulary is in the metadata Recommendation. For making the agreement with a provider formal, see making a CSV a contract with the data provider.