Making a CSV Contract With Your Data Provider in Writing

Every recurring CSV feed is a contract, and most of them are verbal. The provider sends a file every morning. You write a loader that works on the first thirty. On day thirty-one the encoding changes, or a column moves, or the file arrives half-written, and the discussion that follows is about who was supposed to do what. A written contract does not stop things from changing. It turns "your loader broke" into "your file does not match version 3 of the agreement", and it gives both sides a test that settles the question in seconds. This article lists what to put in one, shows a machine-readable version, and enforces it with two small scripts, one on each side of the exchange.

What the contract covers

Five groups of agreements. The first two are what people think of as a contract, and the last three are what they forget.

Format. The things a parser needs: encoding (UTF-8, and whether a byte order mark is allowed; see the encoding series and BOM handling), delimiter, quote character, how a quote inside a field is written, line ending (CRLF or LF), whether there is a header row, whether the last line ends with a newline, and compression. RFC 4180 describes the common form, but a contract should not say "RFC 4180" and stop: the RFC is informational, and its own text leaves choices open.

Content. The columns, in order, by name. For each: what it holds, whether it can be empty, the format (the date is YYYY-MM-DD, in which time zone, or no time at all), the number format (decimal point, no thousands separator, no currency symbol), the allowed values for categories, and which column or combination is the key and must be unique. Say what an empty value means: missing, not applicable or zero are three different facts that a CSV writes the same way. Say what a "null" token looks like if there is one.

Delivery. The file name pattern, with the date in it. The schedule and the cutoff: what time the data is as of and what time the file will be there. The channel (SFTP, object storage, an API) and how you know a delivery is complete. A good answer is a manifest written after the data file, which I cover below. Also what a correction looks like: does a resend replace the previous file with the same name, or arrive under a new one? Is each file the full state or a change since the last one?

Change. How the contract is versioned, how much notice a breaking change needs (30 days is typical, and I put it in the file), and what counts as breaking. My definition: removing, renaming or reordering a column, changing a type or a format, or changing the meaning of a value. Adding a new column at the end can be non-breaking if the consumer says so in advance. Both sides must agree on that sentence before it matters.

Failure. What happens when a file does not conform: the consumer rejects it, tells the provider within an agreed time, and the provider resends within an agreed time. Who is the contact on each side. A contract without this section reads well and does nothing when a delivery goes wrong at 3 a.m.

A contract your code can read

The prose version goes in a document that both parties sign off on. The machine-readable version is the same content as data:

{
  "contract_version": 1,
  "dataset": "customers",
  "file": "customers-YYYYMMDD.csv",
  "encoding": "utf-8",
  "bom": false,
  "delimiter": ",",
  "quote": "\"",
  "line_ending": "\n",
  "header": true,
  "empty_means": "missing",
  "dates": "YYYY-MM-DD, no time, UTC calendar day",
  "numbers": "decimal point, no thousands separator, no currency symbol",
  "key": ["id"],
  "columns": [
    {"name": "id", "pattern": "\\d+", "unique": true},
    {"name": "first_name"},
    {"name": "last_name"},
    {"name": "email", "pattern": "[^@\\s]+@[^@\\s]+\\.[^@\\s]+", "unique": true},
    {"name": "phone", "pattern": "|\\+1\\d{10}"},
    {"name": "phone_unparsed"},
    {"name": "signup_date", "pattern": "|\\d{4}-\\d{2}-\\d{2}"},
    {"name": "balance", "pattern": "|-?\\d+\\.\\d{2}"},
    {"name": "balance_currency", "pattern": "|USD|EUR"},
    {"name": "units", "pattern": "|-?\\d+"},
    {"name": "country", "pattern": "|US|IT|DE|BR"},
    {"name": "status", "pattern": "active|pending|inactive"}
  ],
  "change_notice_days": 30,
  "owner": "[email protected]"
}

The top half holds the format and conventions, the columns list holds the names in order with a pattern for each value (an empty alternative in the pattern, as in |\+1\d{10}, makes the field optional), and unique marks the keys. It ends with the notice period and an owner. This is deliberately a plain file with no schema language. A richer description is possible, such as a Table Schema as covered in the Frictionless article, or the W3C's CSV on the Web metadata. I use JSON here because it keeps the focus on what the contract says, and it can be converted to either of those later.

What matters is that the verifier reads this file. If the rules in the document and the rules in the code are two separate things, they will diverge, and the first time they do is when you most need them to agree.

The provider writes a manifest

The data file alone cannot prove that it is complete. A manifest travels with it and says what the file should be: its name, its size in bytes, its SHA-256, its row count, the contract version and the column list. The provider writes it after the data file, so its presence means the data is finished. This is the provider's script:

"""Provider side: write the manifest that travels with a file. Written last, so its presence means the file is complete."""
import csv, hashlib, json, sys

path, contract_path = sys.argv[1], sys.argv[2]
raw = open(path, "rb").read()
contract = json.load(open(contract_path))
rows = sum(1 for _ in csv.reader(open(path, newline="", encoding="utf-8"))) - 1
manifest = {
    "contract_version": contract["contract_version"],
    "file": path.rsplit("/", 1)[-1],
    "bytes": len(raw),
    "sha256": hashlib.sha256(raw).hexdigest(),
    "rows": rows,
    "columns": [c["name"] for c in contract["columns"]],
}
open(path + ".manifest.json", "w").write(json.dumps(manifest, indent=1) + "\n")
print(json.dumps({k: manifest[k] for k in ("file", "bytes", "rows")}))

python lab/val/make_manifest.py work/val/delivery/customers-20251001.csv lab/val/customers.contract.json
cat work/val/delivery/customers-20251001.csv.manifest.json

{"file": "customers-20251001.csv", "bytes": 17297141, "rows": 193708}
{
 "contract_version": 1,
 "file": "customers-20251001.csv",
 "bytes": 17297141,
 "sha256": "8411365bdfdabc3b40d89f72a4d25c64ea1271ef159f0e6b3fa65bdcbbc23334",
 "rows": 193708,
 "columns": [
  "id",
  "first_name",
  "last_name",
  "email",
  "phone",
  "phone_unparsed",
  "signup_date",
  "balance",
  "balance_currency",
  "units",
  "country",
  "status"
 ]
}

For the customer file: 17,297,141 bytes, 193,708 rows, SHA-256 beginning 8411365bdfdabc3b, contract version 1. It is a small JSON file next to the CSV, named after it. If the provider cannot run a script, the same three numbers from wc -c, sha256sum and a row count in a plain text note are 90% of the value.

The consumer verifies

The consumer's script reads the contract and the manifest, and decides to accept the delivery or reject it, with every reason:

"""Consumer side: accept a delivery only if it matches its manifest and the contract. Exit 0 accepted, 2 rejected."""
import csv, hashlib, io, json, re, sys

SUPPORTED_CONTRACT = 1


def verify(path, contract_path):
    problems = []
    contract = json.load(open(contract_path))
    try:
        manifest = json.load(open(path + ".manifest.json"))
    except FileNotFoundError:
        return ["no manifest next to the file: treat the delivery as unfinished"]
    if manifest["contract_version"] != SUPPORTED_CONTRACT:
        problems.append(f"manifest is for contract version {manifest['contract_version']}, this consumer supports {SUPPORTED_CONTRACT}")
    raw = open(path, "rb").read()
    if len(raw) != manifest["bytes"]:
        problems.append(f"{len(raw)} bytes, manifest says {manifest['bytes']}")
    if hashlib.sha256(raw).hexdigest() != manifest["sha256"]:
        problems.append("sha256 differs from the manifest")
    try:
        text = raw.decode(contract["encoding"])
    except UnicodeDecodeError as e:
        return problems + [f"not {contract['encoding']}: {e}"]
    if "\r" in text:
        problems.append(f"contract line ending is {contract['line_ending']!r}, file has carriage returns")
    rows = list(csv.reader(io.StringIO(text, newline=""), delimiter=contract["delimiter"], quotechar=contract["quote"], strict=True))
    header, body = rows[0], rows[1:]
    names = [c["name"] for c in contract["columns"]]
    if header != names:
        problems.append(f"header {header} differs from the contract {names}")
        return problems
    if len(body) != manifest["rows"]:
        problems.append(f"{len(body)} rows, manifest says {manifest['rows']}")
    wrong = [i + 2 for i, r in enumerate(body) if len(r) != len(names)]
    if wrong:
        problems.append(f"{len(wrong)} rows are not {len(names)} fields wide (first lines {wrong[:3]})")
    seen = {}
    for col in contract["columns"]:
        k = names.index(col["name"])
        bad = [i + 2 for i, r in enumerate(body) if len(r) == len(names) and col.get("pattern") is not None and not re.fullmatch(col["pattern"], r[k])]
        if bad:
            problems.append(f"{col['name']} breaks {col['pattern']!r} in {len(bad)} rows (first lines {bad[:3]})")
        if col.get("unique"):
            vals = [r[k] for r in body if len(r) == len(names)]
            if len(set(vals)) != len(vals):
                problems.append(f"{col['name']} is not unique")
    return problems


if __name__ == "__main__":
    problems = verify(sys.argv[1], sys.argv[2])
    for p in problems:
        print("REJECT:", p)
    print("accepted" if not problems else f"{len(problems)} problems")
    sys.exit(2 if problems else 0)

It checks, in order: that a manifest exists, that its contract version is one the consumer supports, the byte size and the checksum, that the file decodes in the contract's encoding and has the contract's line ending, that the header equals the contract's column list exactly, that the row count equals the manifest's, that every row has the right width, and the pattern and uniqueness rules per column. It does not stop at the first failure when a failure does not make later checks meaningless, which means the report lists everything that is wrong in one go. This is the opposite of what the database does, as the duplicate-keys article shows.

Six deliveries, one outcome each.

The intact delivery:

python lab/val/verify_delivery.py work/val/delivery/customers-20251001.csv lab/val/customers.contract.json; echo "exit status: $?"

accepted
exit status: 0

Accepted, exit status 0.

One byte changed after the manifest was written. I added a Z to a first name on line 5:

python lab/val/verify_delivery.py work/val/delivery/customers-20251002.csv lab/val/customers.contract.json; echo "exit status: $?"

REJECT: 17297142 bytes, manifest says 17297141
REJECT: sha256 differs from the manifest
2 problems
exit status: 2

Rejected twice: the file is 17,297,142 bytes where the manifest says 17,297,141, and the SHA-256 differs. The values all still pass their patterns, which is the point of a checksum. It also tells you what to do next: ask for the file again.

Five rows missing. The file was cut between records, so it is well-formed, with a stale manifest:

python lab/val/verify_delivery.py work/val/delivery/customers-20251003.csv lab/val/customers.contract.json; echo "exit status: $?"

REJECT: 17296676 bytes, manifest says 17297141
REJECT: sha256 differs from the manifest
REJECT: 193703 rows, manifest says 193708
3 problems
exit status: 2

Rejected on size, checksum and row count: 193,703 against 193,708. This is the cut that no structural test can see, as the truncation article shows, and the manifest sees it three ways.

A renamed column. The provider's own manifest was honest, because the provider generated it from the file they sent:

python lab/val/verify_delivery.py work/val/delivery/customers-20251004.csv lab/val/customers.contract.json; echo "exit status: $?"

REJECT: header ['id', 'first_name', 'last_name', 'email', 'phone', 'phone_unparsed', 'signup', 'balance', 'balance_currency', 'units', 'country', 'status'] differs from the contract ['id', 'first_name', 'last_name', 'email', 'phone', 'phone_unparsed', 'signup_date', 'balance', 'balance_currency', 'units', 'country', 'status']
1 problems
exit status: 2

The size and checksum match, so the delivery is intact, but the header says signup where the contract says signup_date. Rejected, with both lists printed so nobody has to diff them. This is the case the contract exists for: a perfectly valid file that is not the file agreed.

No manifest:

python lab/val/verify_delivery.py work/val/delivery/customers-20251005.csv lab/val/customers.contract.json; echo "exit status: $?"

REJECT: no manifest next to the file: treat the delivery as unfinished
1 problems
exit status: 2

The data file is there and the manifest is not, which means either the transfer is in progress or the provider's job died after writing the data. Either way, the delivery is unfinished. This is the main reason the manifest is written last.

A manifest for a newer contract:

python lab/val/verify_delivery.py work/val/delivery/customers-20251005.csv lab/val/customers.contract.json; echo "exit status: $?"

REJECT: manifest is for contract version 2, this consumer supports 1
1 problems
exit status: 2

The provider has moved to contract version 2 and the consumer supports 1. Rejected early, with a message that points at the version and not at forty confusing pattern failures. A version number in the manifest is a few bytes, and it turns an unannounced breaking change into a clear message.

Getting the agreement

You may be the weaker party in the relationship, and a long document is not going to be signed. Start small, and expand it as problems occur.

1. Ask for a sample file and a description of every column, in writing. A sample from the real system, not a hand-built one.

2. Agree on the minimum: file name pattern, encoding, delimiter, header and column list, plus a count and a checksum.

3. Ask for a change notice, with a number of days. Even "we'll tell you by email before we change a column" is enough to start.

4. Offer something back. A report that shows the provider how many of their rows fail which rule is useful to them. Many providers have never seen their own data from the outside.

5. Add to the contract after each incident. The failure that cost you a day is the clause that should have been there, and it is easier to ask for with a concrete example.

When there is no provider to negotiate with, because you are consuming a public data set, write the contract anyway, from what you observe, and pin the version you tested. The verifier will tell you the day it changes.

How to tell it worked

Run the verifier on a good file and on a deliberately broken one for each clause. Give the provider the contract and the sample and ask them to run their next delivery through your verifier before they send it; the first time will fail for reasons that are quick to fix. Keep the accepted manifests, so the history of deliveries is a list of sizes, counts and checksums that you can compare. And keep the contract in version control, with each change recorded alongside the date the provider agreed to it.

What a contract cannot make anyone do

A contract is evidence and a test, not enforcement. If the provider has no obligation to follow it, or no one to ask when it breaks, the verifier will reject files and nothing more will happen. It does not make the data true: a file can match the contract perfectly and report the wrong numbers, and for that you compare with an independent count. And it cannot anticipate everything. Treat each unexpected problem as a missing clause, add it, and keep going. A contract that is revised after each incident ends up as an accurate description of how the data actually breaks, which is the most useful document in the pipeline. The validation guide shows where its checks fit into the rest.