Tailing and Following a CSV That Is Still Being Written

Some CSVs never finish. A log exporter appends a row per event, a sensor writes a line a second, a long job dumps results as it goes. You want to watch it, filter it or feed it to something else while it is still being written. The Unix tools for this are old and good, and each has one habit that will bite you. This article covers following a CSV with tail, the three problems that appear (the header, rotation and partial lines), and a small Python follower that handles them.

The basic follow

tail -f events.csv

tail -f prints the last ten lines and then keeps printing what is appended. Two options matter immediately. -n chooses the starting point, and a leading plus means "from line N":

tail -n +2 -f events.csv     # everything after the header, then new rows
tail -n 0  -f events.csv     # only rows written from now on

The first form is the one for a CSV: it starts at line 2, so the header never appears, and then follows. Use -n 0 when you only care about the future.

Rotation: -f versus -F

tail -f follows the file descriptor it opened. If a log rotator renames events.csv to events.csv.1 and starts a new events.csv, tail -f keeps reading the old, renamed file and prints nothing more. tail -F is shorthand for --follow=name --retry: it follows the name, notices that it now points to a different file, and reopens it. For a CSV that a job rewrites or rotates, always use -F:

tail -n +2 -F events.csv

Note that a rotation brings a new header in the new file. tail -F prints a message when it reopens and then starts from the top of the new file, so the header of every generation appears in your stream. If that matters, filter it out downstream (grep -v '^id,') or have the producer write the header once per file and the consumer remember it.

The half-written line

A writer appends bytes, not lines. The reader may see 1042,2026-10-26T09: before the rest arrives. tail prints what it has, and a downstream parser that treats each chunk as a line will see a broken row. Two defenses: have the producer write each row with one write() call and a trailing newline, which on most local file systems makes the line appear atomically for small rows; and have the consumer refuse to act on text that does not end with a newline.

Here is a follower in Python that does the second:

import csv, io, os, time

def follow(path, poll=0.5):
    with open(path, "r", newline="", encoding="utf-8") as f:
        header = f.readline()
        buf = ""
        while True:
            chunk = f.read()
            if not chunk:
                if os.stat(path).st_size < f.tell():     # truncated: start again
                    f.seek(0); header = f.readline(); buf = ""
                time.sleep(poll)
                continue
            buf += chunk
            *lines, buf = buf.split("\n")                 # the last piece may be partial
            for row in csv.reader([header] + lines):
                if row is not header:
                    yield row

for row in follow("events.csv"):
    print(row)

The key line is *lines, buf = buf.split("\n"): everything before the last newline is complete, and whatever follows stays in buf until more arrives. It does not handle rotation by rename (for that, compare os.stat(path).st_ino to the inode of the open file and reopen), and it assumes each line is one record, which the next section questions.

Buffering: why nothing appears

You wire the stream into a filter and see nothing for a long time:

tail -n +2 -F events.csv | grep ',ERROR,' | cut -d, -f1,4

When a program's output is a pipe and not a terminal, the C library buffers it in blocks of several kilobytes. grep waits until its buffer fills before it writes, so with a slow trickle of rows you wait minutes. The fix is per tool:

tail -n +2 -F events.csv | grep --line-buffered ',ERROR,' | awk -F, '{ print $1, $4; fflush() }'
tail -n +2 -F events.csv | stdbuf -oL cut -d, -f1,4

grep --line-buffered flushes after each line, awk's fflush() does the same, and stdbuf -oL forces line buffering for programs that use the standard C library. It does nothing for programs that have their own buffering, so check each stage. The last stage of a pipeline writing to a terminal is not a problem; the middle stages are.

When a line is not a record

All of the above assumes one record per line. A CSV can have a quoted field containing a line break, and then the follower sees 5,"two and, later, lines",x. Splitting on newlines hands a parser half a record. Two options. Make the producer promise that fields never contain line breaks (replace them with a space or \n text), which is the right answer for a log. Or follow with a stateful parser: feed bytes to csv.reader through an iterator that yields lines, and let the reader pull more lines when it is inside a quote; the reader returns a record only when it has a complete one. The counting article shows how common such fields can be in a file that looks plain.

Working with the stream

Once rows flow, the usual tools apply. awk -F, '$4 == "ERROR"' filters, cut picks columns, and a loop that appends to a file or posts to a service handles the rest. Be careful with -F, and quoted commas: awk splits on every comma. If a field can hold a comma, parse with a CSV-aware tool such as mlr --icsv --ojson or qsv in the pipe, or use the Python follower. For counting, tail -F ... | pv -l -i 5 > /dev/null shows rows per second.

How to tell it worked

Start the follower, append rows from another shell with echo '1,a' >> events.csv, and watch them arrive. Then test the three failure cases on purpose: append a row in two writes (printf '2,b'; sleep 2; echo ',c') and see that the follower waits for the newline; rename the file and create a new one and see that -F reopens it; and put a slow filter in the pipe and see whether lines appear promptly. Run the follower for a day on a quiet file, because the bug that matters is the one that appears at 3 a.m.

When following a file is the wrong design

Following a file is polling dressed up. If the producer can write to a queue, a socket or a database, do that; a consumer of a file has no acknowledgement, no replay after a crash and no backpressure. If you must use a file, make it append-only, record the byte offset you have processed so a restart resumes, and write complete lines atomically. For files that are written once and read later, none of this applies: use the line-counting and splitting tools instead. The large-file hub lists the neighbors of this technique.

Further reading: GNU tail manual; GNU grep manual; Python csv module.