Handling CSV With Comment Lines and Metadata Headers

Open some CSV files and the first lines are not data: a hash sign, a sentence about how the file was produced, a warning not to edit it. Scientific instruments, ledger tools, statistics offices and a surprising number of scripts do this. RFC 4180 has no notion of comments, so a standard reader treats the comment as the header, takes its first words as the column names and shifts everything. The remedy is to remove comment lines before the parser sees them, and the one library option that seems built for this, pandas' comment argument, has a trap that corrupts data without a warning. This article shows both, with output. The variant belongs with the other decorated files in CSV dialects and variants you will run into.

The file

Here is a small export with two comment lines at the top, one at the bottom and a value that contains a hash sign:

# Export generated 2026-10-02 by ledger-tool 4.2
# Do not edit
order_id,customer,note
1001,Acme Ltd,Suite #4
1002,Beta Corp,plain
# trailing note

A person sees the data at once. A strict reader sees a header of one column named # Export generated 2026-10-02 by ledger-tool 4.2, a second line with one field, and a ragged shape. Pandas without options raises a ParserError about the expected number of fields, because the comment has one field and the data lines have three.

The option that looks right

Pandas has a comment argument documented as "character indicating the remainder of the line should not be parsed". It sounds exactly right:

import pandas as pd
df = pd.read_csv("commented.csv", comment="#", dtype=str)
print(df.to_string(index=False))

order_id  customer   note
    1001  Acme Ltd Suite 
    1002 Beta Corp  plain

The comment lines are gone, and so is part of the data: Suite #4 is now Suite with a trailing space. The option does exactly what the documentation says. It ignores the rest of any line from the first #, whether the marker starts the line or sits inside a value. No error was raised and the shape of the frame is right, which makes this a quiet data-corruption bug: street addresses with #, color codes like #ff0000, issue numbers like #1234, hashtags and channel names all lose their tails. The word "line" in the documentation is the trap, since the option applies from the marker to the end of the line, wherever the marker is.

If the marker could appear in any value, never use that option.

Filter whole lines instead

The safe approach removes only lines that begin with the marker, before parsing. In a shell:

grep -v '^#' commented.csv

order_id,customer,note
1001,Acme Ltd,Suite #4
1002,Beta Corp,plain

The three comment lines are gone and the hash inside Suite #4 is untouched. The same in Python, which also lets you keep what you remove:

import io, pandas as pd

comments, data = [], []
with open("commented.csv", newline="", encoding="utf-8") as f:
    for line in f:
        (comments if line.startswith("#") else data).append(line)

df = pd.read_csv(io.StringIO("".join(data)), dtype=str)
print(df.to_string(index=False))
print(comments)

order_id  customer     note
    1001  Acme Ltd Suite #4
    1002 Beta Corp    plain

Suite #4 survived. The list comments holds the three removed lines, which is not a detail to throw away: the first one says when the file was generated and by which program version, which is exactly what you want in the log of a load (automating CSV imports into a database). Parse the date out of it if the format is stable, and treat a changed comment format as a sign that the producer changed.

When a line starting with the marker is data

The line filter assumes that a line beginning with # is never part of a record. That fails when a quoted field spans several lines and one of those lines happens to start with the marker:

id,note
1,"first line
# not a comment, still inside the quoted field
last line"

A raw-line filter deletes the middle line and changes the value. If your data can contain multi-line quoted fields, filter after a real CSV parse, on records, and not before: parse with a reader that understands quoting, then drop records whose first field starts with the marker. That is more work and correct. Check whether your data has such fields by counting the lines against the records (checking row and column counts in a CSV); if the line count equals the record count plus the header, the raw filter is safe.

Comments in the middle, and other markers

Comments are not always at the top. Some tools put a note before each section, and some write a footer ("# End of file, 2 records"). The line filter handles all of them, and the footer is useful as a control: parse the count from it and compare with the rows you loaded. Other markers appear too: // in files that came out of code generators, ; in some configuration-flavored CSVs and % in instrument output. Write the marker into the reader's configuration, as with any other dialect setting, and test with a line that contains the marker in the middle of a value.

Tools differ on what a comment option means, and the differences matter. In the lab container, qsv input --comment '#' removed only lines that start with the marker and left Suite #4 intact, which is the safe semantics. DuckDB 2.0's read_csv(..., comment='#') behaved like pandas: it returned Suite for that row, cutting the value at the hash. Python's csv module has no comment feature, so you filter lines before passing them in. Whatever the tool, write the marker into the reader's configuration like any other dialect setting, test it on a line that contains the marker in the middle of a value, and read the documentation for whether it means "line starts with" or "from here on". Other markers appear too: // in files from code generators, ; in some configuration-flavored CSVs and % in instrument output.

Metadata headers that are not comments

A related shape is a block of key: value lines above the header with no comment marker: Instrument: X200, Run date: 2026-10-02, then a blank line, then the CSV. There is nothing to filter on except position, so read the metadata lines yourself up to the blank line, store them as a dictionary and hand the rest to the CSV reader. If the number of metadata lines varies, find the header by its content (the first line that matches the expected column names) and skip to it. Parsing multi line headers in report style CSV files covers headers that span lines, which is the harder cousin.

When the hash is data

If the hash appears in values and the producer also uses it for comments, the format is ambiguous and the producer should fix it. Ask for the comments to go to a separate file, or for a marker that cannot appear in data. A format where # starts a comment and also appears in values is only readable with a rule about position ("only at the start of a line, outside quotes"), and that rule is yours to write and test.

For feeds you control, do not write comments into the data file at all. Put the export time, the program version and the row count in a manifest next to the file, or in the file name. A CSV that is only data can be read by every tool without special handling, and the metadata gets a proper home. Pandas documents the comment option, with its line-level behavior, in the read_csv reference, and the qsv options are listed in its repository.