How to Detect the Encoding of a CSV File Before You Load It

A file arrives, the accents are wrong, and the first question is what encoding it uses. The honest answer is that nobody wrote it down. A CSV is bytes with no label, so every tool that claims to detect the encoding is inferring it from patterns. This procedure uses four detectors, shows where they disagree on a set of six files, and ends with the check that matters, which is reading the decoded text. For the problem this belongs to, see fixing garbled text in CSV files for good.

What you need

The file, and a little knowledge of what it should contain. A line with a known accented name or a currency sign is the best test, because that is where encodings differ. I generated six files from one table that contains José Müller, São Paulo, a euro price, curly quotes, and in two files Japanese and Polish text: UTF-8, UTF-8 with a BOM, Windows-1252, ISO-8859-1, UTF-16 and Shift_JIS.

"""Write the same small table in the encodings people actually send, so every encoding article starts from the same bytes."""
import codecs
import pathlib

out = pathlib.Path("enc")
out.mkdir(exist_ok=True)

WEST = "id,name,city,note\n1,José Müller,São Paulo,price €4.50\n2,Zoë Dupré,Zürich,“smart quotes”\n3,Åsa Öberg,Malmö,café\n"
LATIN = "id,name,city\n1,José Müller,São Paulo\n2,Zoë Dupré,Zürich\n3,Åsa Öberg,Malmö\n"
MIXED = WEST + "4,田中 太郎,東京,日本語\n5,Łukasz Nowak,Kraków,zażółć\n"
JA = "id,name,city\n1,田中 太郎,東京\n2,山田 花子,大阪\n3,佐藤 次郎,京都\n"

files = {
    "utf8.csv": MIXED.encode("utf-8"),
    "utf8-bom.csv": codecs.BOM_UTF8 + MIXED.encode("utf-8"),
    "cp1252.csv": WEST.encode("cp1252"),
    "latin1.csv": LATIN.encode("latin-1"),
    "utf16.csv": MIXED.encode("utf-16"),
    "shift_jis.csv": JA.encode("shift_jis"),
}
for name, data in files.items():
    (out / name).write_bytes(data)
    print(f"{name:14} {len(data):4} bytes")

Step 1: let file and uchardet guess

file -i reads a few kilobytes and reports a character set. It is quick and it is conservative, so for anything it cannot place it says unknown-8bit:

cd enc && file -i *.csv

ascii.csv:     text/csv; charset=us-ascii
cp1252.csv:    text/csv; charset=unknown-8bit
latin1.csv:    text/csv; charset=iso-8859-1
shift_jis.csv: text/csv; charset=unknown-8bit
utf16.csv:     text/plain; charset=utf-16le
utf8-bom.csv:  text/csv; charset=utf-8
utf8.csv:      text/csv; charset=utf-8

It identified UTF-8 with and without the BOM, ISO-8859-1 for the Latin file and UTF-16. It did not name the Windows-1252 or Shift_JIS files, which is the correct response when a tool cannot tell. Note also that it called the UTF-16 file text/plain, because a CSV in UTF-16 does not look like CSV to a tool that scans bytes.

uchardet is the Mozilla detection library as a command. It names an encoding for each file:

cd enc && for f in *.csv; do printf '%-14s %s\n' $f "$(uchardet $f)"; done

ascii.csv      ASCII
cp1252.csv     WINDOWS-1252
latin1.csv     ISO-8859-1
shift_jis.csv  SHIFT_JIS
utf16.csv      UTF-16
utf8-bom.csv   UTF-8
utf8.csv       UTF-8

All six correct, including the two that file declined. That does not make it infallible; it makes it a good first answer for files with enough non-ASCII text to analyze.

Step 2: ask the Python detectors, and read the confidence

charset-normalizer and chardet are what libraries like requests and pandas pipelines often use. Run on the same files:

cd enc && python - <<'PY'
import glob
from charset_normalizer import from_bytes
import chardet
print(f"{'file':14} {'charset-normalizer':24} {'chardet':28}")
for f in sorted(glob.glob("*.csv")):
    raw = open(f, "rb").read()
    n = from_bytes(raw).best()
    c = chardet.detect(raw)
    print(f"{f:14} {n.encoding if n else None!s:24} {c['encoding']} ({c['confidence']:.2f})")
PY

file           charset-normalizer       chardet                     
ascii.csv      ascii                    ascii (1.00)
cp1252.csv     cp1250                   Windows-1252 (0.06)
latin1.csv     cp1250                   Windows-1252 (0.05)
shift_jis.csv  cp932                    CP932 (0.03)
utf16.csv      utf_16                   UTF-16 (1.00)
utf8-bom.csv   utf_8                    UTF-8-SIG (1.00)
utf8.csv       utf_8                    utf-8 (0.99)

Two things stand out. charset-normalizer returned cp1250 for the Windows-1252 file and the Latin 1 file, which is wrong: cp1250 is the Central European code page, and decoding with it would turn é into a different letter. It got the others right. chardet named Windows-1252 for both Western files, which is a defensible answer (Windows-1252 is a superset of Latin 1 for printable characters), but with a confidence of 0.06 and 0.05. It named CP932 for the Shift_JIS file, which is Microsoft's variant and decodes the same text, with a confidence of 0.03.

A confidence under 0.5 is the detector telling you it does not know. The six files here are tiny, so that is partly the sample size. The point is to use the number, not only the name.

Step 3: decode a sample and read it

No detector replaces this. Pick the answer, decode the file with it and look at the lines with accents. If José and São Paulo read correctly, the encoding is right for those bytes. If you see José, the file is UTF-8 and was decoded as something else, and if you see Jos followed by a replacement character the file is not UTF-8. The next article, why a CSV shows strange characters instead of accents, maps each symptom to its cause.

Where two encodings both decode without error and give different letters, only your knowledge of the data decides. Windows-1252 and ISO-8859-1 differ only in 32 positions, mostly punctuation and the euro sign, so a file of French names reads the same in both and a file with curly quotes does not. Windows-1252 and cp1250 differ in the letters they place at the same byte, so reading the wrong one turns Western accents into Central European ones, and the text looks plausible and is wrong.

What the detectors are looking at

None of them read a label. uchardet is the detection library from the Mozilla project, and it scores each candidate encoding by how plausible the byte patterns are: which byte pairs occur, whether the sequences are legal in that encoding, and how common the resulting letters are in the languages it knows. chardet is a port of the same idea and charset-normalizer takes a different route, decoding with many candidates and measuring how coherent the decoded text looks. The documentation for charset-normalizer describes that approach and its confidence measure.

That explains the pattern in my table. A tiny file gives a statistical method very little to count, and two encodings that differ in a handful of positions, like Windows 1252 and Latin 1, look the same to it. It also explains why the Japanese file was easy for uchardet and hard for chardet: Shift_JIS has a distinctive byte structure, and the evidence is stronger there than for accents. When a file is large, mostly text and in one language, expect better answers than I got here.

A web standard shows how deep the Windows 1252 and Latin 1 confusion goes. The WHATWG Encoding Standard maps the label iso-8859-1 to Windows 1252 for browsers, because so much content labeled Latin 1 is really Windows 1252. A browser will therefore render your Latin 1 file with a euro sign where bytes 80 would have been control characters. Many programs do not follow that convention, so the same file reads differently in different tools.

Which detector to trust

For a quick answer, uchardet followed by a decode check. For automation inside a Python pipeline, charset-normalizer with a check for the result being one of the encodings you expect from that sender, so a surprising answer fails loudly. For anything marked as UTF-8 or UTF-16 with a BOM, believe the marker: it is written into the file.

I do not use detection as a way to load files without thinking. If a sender always uses the same program, the encoding is a fact about that program, and writing it down is cheaper than guessing every time. A short note in the data contract, as in the contract article, turns a detection problem into configuration.

When no detector can know

A file that is all ASCII is valid in nearly every encoding, so a detector reporting ASCII is correct and uninformative. A file with a handful of accented characters has too little evidence for a statistical guess, which is what the 0.03 to 0.06 scores show. A file that mixes encodings, because two exports were concatenated, decodes correctly under no single choice. And a file that has been through an earlier conversion may be valid UTF-8 and still wrong, because the text was already corrupted; that is mojibake repair, not detection.

In those cases ask the sender which program produced the file and with which locale settings, or compare against a known record in the source system. A detector can narrow the list. The person who ran the export can end it.