What RFC 4180 Says and Does Not Say About CSV Files
RFC 4180 is a four page memo, not a standard. Read its seven rules, the grammar quirks that surprise people, and how real files and readers drift from it.
Anyone who has argued about CSV in a pull request has seen RFC 4180 quoted as if it were the law. It is a four page memo from October 2005 whose own first paragraph says it "does not specify an Internet standard of any kind". That does not make it useless. It is the closest thing the format has to a shared description, and reading it takes ten minutes. This article goes through what it actually says, the two or three things in it that surprise people, and how far real files and real readers stray, as measured in the CSV format guide.
The status line most people skip
Before the rules, the header of the document tells you what kind of document it is.
curl -s https://www.rfc-editor.org/rfc/rfc4180.txt | sed -n '/^Category/p;/^ This memo provides/,/standard of any kind/p'Category: Informational October 2005
This memo provides information for the Internet community. It does
not specify an Internet standard of any kind. Distribution of this"Informational" is an IETF category. It means the memo records something useful to the community, and it did not go through the process that produces a standard. The author, Yakov Shafranovich, wrote it to document existing practice and to register the text/csv media type with the IANA media types registry. Registration is why a web server can serve a .csv file with the right content type. It is also nearly everything the RFC binds anyone to.
The seven rules in plain language
Section 2 of the RFC is a numbered list of seven rules. I have not shortened the sense of any of them.
Each record sits on its own line, and the line break is CRLF.
The last record may or may not end in a line break.
The first line may be a header with the same shape as a record, and whether it exists should be signalled by the
headerparameter of the media type.Fields are separated by commas, every line should have the same number of fields, spaces count as part of the field, and the last field must not be followed by a comma.
A field may be wrapped in double quotes. If it is not wrapped, it may not contain a double quote.
A field that contains a line break, a double quote or a comma should be wrapped.
Inside a wrapped field, a double quote is written as two double quotes.
That is the complete rule set, and the first thing it tells you is how small the format is. There is no rule about encoding, about what an empty field means, about numbers, dates, or booleans, and no rule that allows a backslash escape. The RFC also borrows a line of advice from an older one: be conservative in what you send and liberal in what you accept. That is guidance for implementors, not a feature of the format.
The grammar, and the surprise inside it
The RFC also gives an ABNF grammar. This is the part nobody reads, and it holds the most interesting detail.
curl -s https://www.rfc-editor.org/rfc/rfc4180.txt | grep -E '^ (file|header|record|name|field|escaped|non-escaped|COMMA|DQUOTE|TEXTDATA|CRLF) =' file = [header CRLF] record *(CRLF record) [CRLF]
header = name *(COMMA name)
record = field *(COMMA field)
name = field
field = (escaped / non-escaped)
escaped = DQUOTE *(TEXTDATA / COMMA / CR / LF / 2DQUOTE) DQUOTE
non-escaped = *TEXTDATA
COMMA = %x2C
DQUOTE = %x22 ;as per section 6.1 of RFC 2234 [2]
CRLF = CR LF ;as per section 6.1 of RFC 2234 [2]
TEXTDATA = %x20-21 / %x23-2B / %x2D-7ERead the last line. TEXTDATA is the set of characters allowed in a field without wrapping it in quotes, and its definition is %x20-21 / %x23-2B / %x2D-7E. Those hex ranges are printable ASCII with two holes: %x22, the double quote, and %x2C, the comma. Anything below the space character is out, which means a tab is out. Anything above %x7E is out, which means every accented letter, every currency symbol other than the dollar sign and every character in a non-Latin script is out.
I checked what that means with a short script that asks whether a character is allowed in an unquoted field:
python - <<'PY'
# Which characters does the RFC grammar allow in an unquoted field? TEXTDATA = %x20-21 / %x23-2B / %x2D-7E
allowed = set(range(0x20, 0x22)) | set(range(0x23, 0x2C)) | set(range(0x2D, 0x7F))
for label, ch in [("letter a", "a"), ("space", " "), ("comma", ","), ("double quote", '"'), ("tab", "\t"),
("e acute", "é"), ("euro sign", "€"), ("CJK", "漢")]:
print(f"{label:13} U+{ord(ch):04X} allowed unquoted: {ord(ch) in allowed}")
PYletter a U+0061 allowed unquoted: True
space U+0020 allowed unquoted: True
comma U+002C allowed unquoted: False
double quote U+0022 allowed unquoted: False
tab U+0009 allowed unquoted: False
e acute U+00E9 allowed unquoted: False
euro sign U+20AC allowed unquoted: False
CJK U+6F22 allowed unquoted: FalseTaken literally, the grammar says Zürich cannot appear in a CSV field unless it is wrapped in quotes, and even then the grammar only lists TEXTDATA inside the quotes, so it is silent about the ü there too. Nobody reads it that way. The text/csv registration in the same document has an optional charset parameter and notes that common usage is US-ASCII, with other IANA character sets allowed. The sensible reading is that the 2005 grammar was written for ASCII practice, and character encoding was left to the media type parameter that almost no file carries.
Running the rules as a program
The seven rules that a program can verify are small enough to put in a script. This one checks the record separator, the characters in unquoted fields, and the field counts. It is not a full validator, since it does not know whether row one is a header, but it is enough to see which rules real files break.
"""Check a file against the rules of RFC 4180 section 2 that a program can check. Exit status 1 on any finding."""
import csv
import io
import sys
TEXTDATA = set(range(0x20, 0x22)) | set(range(0x23, 0x2C)) | set(range(0x2D, 0x7F))
def findings(raw: bytes) -> list[str]:
out = []
text = raw.decode("utf-8")
in_quotes, lf_only, crlf, bad = False, 0, 0, {}
for i, ch in enumerate(text):
if ch == '"':
in_quotes = not in_quotes
elif ch == "\n" and not in_quotes:
if i > 0 and text[i - 1] == "\r":
crlf += 1
else:
lf_only += 1
elif ch not in ",\r\n" and ord(ch) not in TEXTDATA and not in_quotes:
bad.setdefault(ch, i)
if lf_only:
out.append(f"{lf_only} record separator(s) are LF, the RFC says CRLF")
for ch, pos in bad.items():
out.append(f"U+{ord(ch):04X} at offset {pos} is outside TEXTDATA in an unquoted field")
widths = {len(r) for r in csv.reader(io.StringIO(text, newline=""))}
if len(widths) > 1:
out.append(f"records have different field counts: {sorted(widths)}")
if in_quotes:
out.append("a quoted field is never closed")
return out
if __name__ == "__main__":
status = 0
for name in sys.argv[1:]:
f = findings(open(name, "rb").read())
print(f"{name}: {'conforms' if not f else str(len(f)) + ' finding(s)'}")
for line in f:
print(" ", line)
status |= bool(f)
sys.exit(status)Three files went through it. The first is an ASCII file written with CRLF, which is what the RFC describes. The second is the awkward orders-awkward.csv from the guide, written with Unix line endings and one name that starts with Ō. The third has a row with one field where the others have two.
python rfc_check.py rfc-clean.csv orders-awkward.csv rfc-ragged.csvrfc-clean.csv: conforms
orders-awkward.csv: 2 finding(s)
5 record separator(s) are LF, the RFC says CRLF
U+014C at offset 70 is outside TEXTDATA in an unquoted field
rfc-ragged.csv: 1 finding(s)
records have different field counts: [1, 2]The awkward file breaks two rules, the line ending and the character set. In every reader I tried (Python, pandas, DuckDB, SQLite, PostgreSQL) it loads, and the records come out right. This is the practical meaning of "informational". A file can be off-spec in two ways and be perfectly usable, and a file can obey the RFC and still mean the wrong thing, because the RFC says nothing about the things that go wrong in practice.
Where the readers side with the RFC
The valid cases in the RFC are the cases every reader agrees on. Quoted commas, doubled quotes, embedded line breaks, a missing final newline, and spaces kept as part of the field all passed in all five readers in the conformance run. The rules that readers treat as advice are the ones phrased with "should".
Rule 4 says each line should have the same number of fields. The tests I ran with one row too long and one row too short showed five different policies. Python returned both rows unchanged. pandas rejected the long row and padded the short one. PostgreSQL rejected both. SQLite imported both with a warning. DuckDB returned one row of three values for the long file and quietly dropped its first line, and read the short file as a single column of text. The word "should" in the rule is where those five policies come from.
Rule 5 says an unwrapped field may not contain a double quote. A line like x"y,z breaks it. Python, pandas, DuckDB and SQLite accepted it and kept the quote as data. PostgreSQL refused with unterminated CSV quoted field. Both policies can be defended, and only one of them tells you there is a problem.
What the RFC leaves out
The omissions are the reason the format is hard in practice. There is no rule for the character encoding, so nothing says whether the bytes C3 A9 are é or two letters from a legacy code page. There is no rule for an empty field, so ,, and ,"", have the same standing as far as the document goes. There is no escaping with a backslash, and no way to write a comment. There is no type system, so 007 might be seven, a part number or a string. There is no marker for the end of the file, so a truncated export is valid. There is no rule for a blank line, which Python reads as an empty record, pandas skips, and PostgreSQL rejects.
Each of those gaps is covered by an article in this cluster. Start with delimiters, quoting and line endings, because those are the pieces the RFC does describe and readers still handle differently.
Fragment identifiers and the W3C documents
Two other documents are worth knowing about because they try to fill some of the gaps. RFC 7111 defines URI fragment identifiers for text/csv, so a link can point at #row=5-7 or #col=2 or a single #cell=4,3. It is a sensible spec, and I have not seen a CSV reader or a spreadsheet that honors it. The W3C CSV on the Web primer goes further. It describes a JSON metadata file that sits next to the CSV and states the column names, datatypes, delimiter, null convention and encoding. That is the right shape for the problem, a contract outside the file, and it is little used outside open data portals.
When the RFC is the wrong yardstick
If you are building a parser for files from the wild, the RFC is a floor, not a target. Accept LF and CRLF and bare CR, accept UTF-8 with and without a byte order mark, and accept a missing final newline, because real exporters do all of those. If your reader insists on CRLF, it will reject most files written on a Mac or a Linux server.
If you are writing a file for Excel, the RFC will not help you. Excel's behavior around encodings, leading zeros and list separators is not described in it, and the article on opening CSV files in Excel without damage covers what a file needs to open cleanly. If you are exchanging files with a bank, a government portal or a customer's import tool, the receiving system's specification is the contract, and it may require a semicolon, a fixed column order or Windows-1252, any of which the RFC does not mention. If you need to guarantee types and nulls, use the metadata approach above or a format that carries a schema, not a stricter reading of a 2005 memo.
For a quick way to see which parts of your own file stray from the RFC, run the checker above and then fix whichever findings matter to your reader. Most of the time that is nothing, and that is the point.
