Detecting Encoded Payloads Hidden in CSV Fields Fast

A CSV column called notes or comment or description can hold anything a person or a program was willing to type. Most of the time that is "Called about invoice 42". Occasionally it is a base64 blob that a script on the other side is going to decode, a snippet of HTML that a report will paste into a page, or a formula that a spreadsheet will run. None of these is dangerous in the file. They become dangerous when something downstream treats the field as more than text. This article gives a short scanner that finds the common shapes before the file is imported, rendered or shared, and explains how to read its output without panicking. It is the practical companion to the encoded payloads section of CSV security risks that catch teams out.

What to look for

The patterns are shapes, not signatures. A base64 string has a limited alphabet and a length that is a multiple of four once padded. Percent-encoded text has % followed by two hex digits, several times. Markup has angle brackets around known tag names, or a javascript: URL, or an event attribute like onerror=. A formula starts with one of four characters. A control character is a byte below space that is not tab, newline or carriage return, and it has no business in a name or a note. And a field of thousands of characters in a column that should hold sentences is suspicious on length alone.

Each shape has an innocent use. A support ticket may quote a URL-encoded query string. A column of file attachments may hold base64 on purpose. So the output of a scan is a list of places to look, and a human reads it.

The scanner

The script reads a CSV with the standard reader and applies each pattern to every field, printing the line number and column name of each hit:

import csv, re, sys

CHECKS = {
    "base64 blob": re.compile(r"^[A-Za-z0-9+/]{40,}={0,2}$"),
    "percent-encoded": re.compile(r"(%[0-9A-Fa-f]{2}.*){3,}"),
    "html or script": re.compile(r"<\s*/?\s*(script|iframe|img|svg|object)\b|javascript:|onerror\s*=", re.I),
    "formula start": re.compile(r"^[=+\-@]"),
    "control character": re.compile(r"[\x00-\x08\x0b\x0c\x0e-\x1f]"),
}
MAX_FIELD = 500

with open(sys.argv[1], newline="", encoding="utf-8") as f:
    reader = csv.reader(f)
    header = next(reader)
    for line_no, row in enumerate(reader, start=2):
        for col, value in zip(header, row):
            hits = [name for name, rx in CHECKS.items() if rx.search(value)]
            if len(value) > MAX_FIELD:
                hits.append(f"long field ({len(value)} chars)")
            if hits:
                print(f"line {line_no} {col}: {', '.join(hits)}")

Here is the test file, with two ordinary notes and four planted ones:

id,name,note
1,Ada,Called about invoice 42
2,Bob,"<img src=x onerror=alert(1)>"
3,Cy,aGVsbG8gd29ybGQgdGhpcyBpcyBhIGxvbmcgYmFzZTY0IGJsb2IgaGlkZGVuIGhlcmU=
4,Di,%3Cscript%3Ealert%281%29%3C%2Fscript%3E
5,Ed,=cmd|' /C calc'!A0
6,Flo,Prefers email

python3 scan_payloads.py notes.csv

line 3 note: html or script
line 4 note: base64 blob
line 5 note: percent-encoded
line 6 note: formula start

All four planted fields are reported with their line and column, and lines 2 and 7 are silent. The line numbers count the header as line 1, and a file with quoted newlines would show record numbers that differ from physical lines, so use the reader's line_num if you need the physical position (checking row and column counts in a CSV explains the difference).

The base64 string in line 4 decodes to a harmless sentence. A scanner cannot know that without decoding, and decoding unknown input is its own risk, so the script flags the shape and stops. If you want to look inside, decode in a sandbox, as text, and print it, never evaluate it.

Tuning for your data

Run the scanner on a known-good file from the same source first and count the hits. A column of product codes may be mostly long alphanumeric strings that match the base64 pattern, and a log column may carry percent-encoded URLs by design. Adjust per column: apply the checks only to free-text columns, exempt columns where the shape is expected, and raise the minimum base64 length until the noise stops. The cost of a false positive is a minute of reading. The cost of a pattern so loose that people ignore the output is the scan being turned off.

Add the checks you need for your context. If the file feeds a SQL tool, flag a single quote followed by a keyword. If the file renders in a report, flag any angle bracket at all. If the consumer is a shell script (a bad idea), flag backticks, dollar signs and semicolons. A narrow, specific scanner for one pipeline beats a general one.

What to do with a hit

Do not delete the row silently, which hides evidence and loses data. Move flagged rows to a quarantine file with their line numbers, load the rest, and send the quarantine to whoever owns the source. Record the count in the job's log. For a feed that has never had a hit and suddenly has twenty, the change is itself the signal, and someone should ask why.

For a hit that is a real attack in progress, such as a script tag in a name field of a public sign-up form, the work moves upstream: validate the field on entry, and check where else the value has already been rendered.

Escape on output, whatever the scan says

Detecting is a second line. The first is to treat every field as inert text at every destination. Render into HTML with an escaping function and never concatenate. Pass values to SQL as parameters. Write to a spreadsheet with the formula escape (escaping formula characters in a CSV export). Put a field into a log line with newlines neutralized, so a value cannot forge a second log entry. If every sink escapes, a payload in the file is just a funny string. The scanner exists for the case where a sink was missed.

It also pairs with the validation you apply at load time: a column declared as a name, a date or a country code should reject anything outside its pattern, and most payloads fail that long before a payload scanner is needed (validating a CSV file).

What a scanner cannot see

A scanner cannot see an encoding it was not written for: double base64, hex, a custom cipher, text split across several fields or several files and assembled later. It cannot tell a harmless string from a harmful one without knowing what the next program does with it. And it cannot see what is not in the file: a field that is clean here and becomes dangerous only when joined with another table.

It also cannot protect you from malware inside a format that is not really a CSV. A file called data.csv that is a spreadsheet or an archive fails the parse before the scan, and a strict reader should reject it up front (limiting file size and row count for CSV uploads covers the shape checks that come first). For anything beyond these checks, the authority is the OWASP guidance on CSV injection and on file upload, and a security review of the pipeline the data goes into.