Open a package.json and try to add a note above a dependency. A line starting with // makes the file fail to parse, and npm tells you the JSON is invalid. Open tsconfig.json in the same project and the same line works, because that file is not JSON at all, it is a relative called JSONC that happens to have the same extension. The difference confuses everyone at some point, and the reason is a design decision from the early days of the format that has never been reversed: standard JSON has no comments, and every strict parser rejects them. This article explains why, confirms it with five parsers, and compares the ways people live with it, from a convention in the data to a different format altogether. The wider picture of what parsers accept is in what JSON allows and what parsers quietly accept.
The input is two lines, a comment and an object:
\n// note\n{\"a\": 1}Every parser I tried rejected it, and each said so in its own words. Python raised JSONDecodeError: Expecting value: line 1 column 1 (char 0), pointing at the first /. Node's JSON.parse said Unexpected token '/' and quoted the input as not valid JSON. PHP's json_decode returned an error code 4, Syntax error. jq 1.6 reported Invalid numeric literal at line 1, column 3. DuckDB's json_valid returned false. A comment placed after the value fails too: {\"a\": 1} // note produced \"Extra data\" in Python, because the parser finished the object and then found more text. Block comments, /* ... */, fail the same way.
There is no parser setting that makes a standards-following parser accept comments, and no accident that makes it happen: the grammar has no place for them. Some tools do accept them, by design, and that is why the confusion arises: an editor that parses settings.json leniently will show no error, and a different program reading the same file will.
Douglas Crockford, who specified the format, has explained that he took comments out of the first versions after noticing people using them to hold parsing directives, instructions to the parser that would make the same document mean different things to different readers. Since the format's value was that every parser reads the same thing, he removed the feature that could break that. Data exchange formats carry data, he argued, and a comment is not data.
\nYou can disagree with that for configuration files and many people do, and the decision is also old, widely deployed and not going to change. The grammar in RFC 8259 and ECMA-404 does not have comments, and a parser that supports them is supporting an extension.
\nPeople who need notes in JSON have found several approaches, each with trade-offs.
\nA comment key. Put the note in a field: {\"_comment\": \"keep in sync with the staging file\", \"timeout\": 30}. It works in every parser, because it is ordinary data. The costs: the note is now part of the document, so every consumer receives it and must ignore it; a strict schema with additionalProperties: false rejects it; JSON has no way to attach a comment to a particular value, only to an object; and duplicate _comment keys in one object overwrite each other, since the last wins. Use a name that cannot collide with real fields and be consistent. Some tools define one: // as a key name is used informally in some package files.
**JSON Schema's $comment.** Schemas, which are themselves JSON, have a keyword for this: $comment is ignored by validators and exists for human notes, from draft-07 onward. Inside a schema file, it is the sanctioned place for a comment, and unlike _comment it is part of the vocabulary. It does not help for data files.
Strip comments before parsing. A tool such as jsmin or a regular expression removes // and /* */ and passes the rest to a real parser. It lets you keep comments in a file you author and still hand strict JSON to the consumer. The danger is the regular expression: a naive one removes // inside a string such as \"http://example.com\", which silently corrupts a URL. Use a real comment-stripping library that understands strings, and test it on a value containing //.
Use a format that has comments. If the file is configuration that people edit, JSON was the wrong container. JSONC, \"JSON with comments\", is what VS Code uses for its settings files and what TypeScript uses for tsconfig.json: standard JSON plus // and /* */ comments, and often tolerated trailing commas. JSON5 adds comments, trailing commas, unquoted keys, single quotes and more. YAML and TOML have comments as a first-class feature and suit configuration well. JSON versus JSON5 versus JSONC and when to use each compares them.
A short guide. If the document is exchanged between programs and a human rarely reads it, do not add comments: put the explanation in the documentation or the schema, which can be much longer than a note. If the file is configuration edited by hand, use a format with comments and parse it with a parser for that format, so that \"lenient\" is a contract and not an accident. If you must annotate JSON that has to remain standard, say a fixture or a feature file consumed by several tools, use a comment key with a reserved name, document it and have every consumer drop it.
\nAvoid half-measures. A .json file containing comments that works in your editor and fails in CI is a trap for the next person, because the extension says one thing and the content another. Name the file by what it is (.jsonc, .json5) and let tools pick the right parser from the name. If your project uses a loader that strips comments (some bundlers and test runners do), make sure the production path uses the same one, since a config with comments read by the lenient dev loader and the strict production parser is a deployment failure waiting to happen.
Many comment needs disappear if the file is generated. A script that writes JSON from a source with comments, a YAML file, a template, a spreadsheet, can keep the notes in the source and emit clean JSON. The generated file says at the top of its documentation, not its contents, that it is generated and where from. That arrangement also fixes the other editing problems of JSON, trailing commas and quoting, because nobody edits the output (trailing commas in JSON and why parsers reject them).
\nA comment key is the wrong choice in a few cases. When the consumer is a strict schema, the key fails validation, and you have to teach the schema about it, which spreads the workaround. When the document is signed or hashed, the comment becomes part of what is signed, and a changed note changes the hash. When the file is large and machine-generated, notes in every object bloat it. And when the note is actually documentation of a field's meaning, it belongs in a schema or a README, where it can be found, versioned and translated. The grammar is in RFC 8259, the $comment keyword in the JSON Schema reference, and the JSONC convention in VS Code's documentation of JSON with comments.
The first time you use PostgreSQL for a project, it is tempting to connect as postgres and create tables. It works. It also means every program you write has the keys to the whole server, and the permission errors that would show up in production never show up on your machine. The better habit costs three statements: a role for the project, a database that role owns and, if you like, a schema inside it. This article walks through them, explains roles and schemas in the PostgreSQL sense, and covers the permission change in version 15 that surprises people who learned the older behavior. It is one of the setup steps in setting up PostgreSQL for development without fighting it and comes after installing PostgreSQL on macOS, Linux and Windows.
PostgreSQL has one concept for accounts and groups: the role. A role may have the LOGIN attribute (so it can connect), may own objects, may have privileges and may be a member of other roles. CREATE USER is an alias for CREATE ROLE ... LOGIN, and CREATE GROUP is an alias for CREATE ROLE without login. You will see all three in tutorials; they create the same thing with different defaults.
A superuser role bypasses all permission checks. The role named postgres is a superuser created at installation. Use superuser roles for administration only.
You need an account that can create roles. Depending on the platform, that is sudo -u postgres psql (Debian, Red Hat family), psql postgres (Homebrew, where your own account is the superuser) or the account and password you chose in the Windows installer or the POSTGRES_USER of a Docker container.
sudo -u postgres psqlAt the postgres=# prompt you can create roles. The # signals a superuser; an ordinary role shows >.
CREATE ROLE shop LOGIN PASSWORD 'dev-only-password';LOGIN allows connections. The password is stored as a hash (SCRAM-SHA-256 by default in current versions). Writing the password in a statement puts it in your psql history and possibly in the server log if statement logging is on. To keep it out, create the role without a password and set it with the \\password meta-command, which prompts and sends the hash:
CREATE ROLE shop LOGIN;\n\\password shopUseful attributes at creation: CREATEDB lets the role create databases (handy for a test runner), CREATEROLE lets it create other roles (avoid for application roles), NOSUPERUSER is the default, CONNECTION LIMIT 20 caps simultaneous sessions, and VALID UNTIL '2027-01-01' makes a password expire. Do not give application roles SUPERUSER.
List roles with \\du:
List of roles\n Role name | Attributes\n-----------+------------------------------------------------------------\n postgres | Superuser, Create role, Create DB, Replication, Bypass RLS\n shop |CREATE DATABASE shop OWNER shop;The owner has full control of the database's contents and can drop it. One database per project is the norm, and it isolates data and settings. Clients connect to a specific database, and a connection cannot query across databases without extra tooling.
\nYou can add ENCODING 'UTF8' and a locale; the defaults from template1 are usually right. Check with \\l:
List of databases\n Name | Owner | Encoding | Locale Provider | Collate | Ctype\n-----------+----------+----------+-----------------+-------------+-------------\n postgres | postgres | UTF8 | libc | en_US.UTF-8 | en_US.UTF-8\n shop | shop | UTF8 | libc | en_US.UTF-8 | en_US.UTF-8Leave the superuser session (\\q) and connect as the project role.
psql -h localhost -U shop -d shopNote the -h localhost. Without it, psql uses the local socket, where Debian and Red Hat family systems apply peer authentication by default and will reject the role shop unless your operating system user is also shop (fixing peer authentication failed for user postgres explains why). Over TCP, the password applies.
Check who you are and where:
\nSELECT current_user, current_database(), current_schema();A schema is a namespace inside a database: a container for tables, views, functions and types. The same table name can exist in two schemas. Every database starts with a schema called public. Tables are referred to as schema.table, and unqualified names are resolved through the search path.
SHOW search_path;\n-- \"$user\", publicThe default search path first looks for a schema named after the current role (shop), then public. If a schema shop exists, unqualified CREATE TABLE orders creates the table there, because it is the first schema in the path that exists.
Create a schema owned by the project role:
\nCREATE SCHEMA app AUTHORIZATION shop;\nALTER ROLE shop SET search_path = app, public;Reconnect for the setting to apply (role-level settings apply to new sessions). Now CREATE TABLE orders (...) in a shop session goes into app. Using a named schema keeps your objects apart from extensions' objects in public and gives you something to grant permissions on. For a single-application database, public is fine, with the caveat below.
Before version 15, every role could create tables in the public schema of any database it could connect to. That was convenient and a security weakness: a shared database invited objects from anyone. From version 15 the privilege is removed: CREATE on public is no longer granted to PUBLIC (everyone), and the database owner owns public.
The consequence for beginners: if you create a role, connect to a database you do not own, and run CREATE TABLE t (...), you see:
ERROR: permission denied for schema publicThree ways to resolve it. Make the role the owner of the database (so it owns public, as in CREATE DATABASE shop OWNER shop). Create the role its own schema and use that. Or, as a superuser or the owner, grant the privilege deliberately:
GRANT USAGE, CREATE ON SCHEMA public TO shop;Prefer the first two. If you upgraded a database from an older version, the old permissive grant may still be in place; check with \\dn+.
Roles can read the project's data without owning it. A read-only role for reporting:
\nCREATE ROLE reporting LOGIN;\nGRANT CONNECT ON DATABASE shop TO reporting;\nGRANT USAGE ON SCHEMA app TO reporting;\nGRANT SELECT ON ALL TABLES IN SCHEMA app TO reporting;\nALTER DEFAULT PRIVILEGES FOR ROLE shop IN SCHEMA app GRANT SELECT ON TABLES TO reporting;The first three give the role entry; GRANT SELECT ON ALL TABLES covers tables that exist now; ALTER DEFAULT PRIVILEGES covers tables the owner creates later, which is the step people forget. Check privileges with \\dp (table privileges) and \\ddp (default privileges).
In production, one role often owns the schema and runs migrations, and a different role with narrower rights serves the application: it can read and write rows but cannot alter tables. Rehearse that in development:
\nCREATE ROLE shop_migrator LOGIN;\nCREATE ROLE shop_app LOGIN;\nALTER SCHEMA app OWNER TO shop_migrator;\nGRANT USAGE ON SCHEMA app TO shop_app;\nGRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA app TO shop_app;\nALTER DEFAULT PRIVILEGES FOR ROLE shop_migrator IN SCHEMA app\n GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO shop_app;\nGRANT USAGE ON ALL SEQUENCES IN SCHEMA app TO shop_app;Running the application as the narrow role locally catches the \"permission denied for table\" error before a deployment does.
\nTo remove a development setup: drop the database first, then the role. A role cannot be dropped while it owns objects or holds privileges; DROP OWNED BY shop; removes its objects and revokes its grants in the current database, then DROP ROLE shop;.
DROP DATABASE IF EXISTS shop;\nDROP ROLE IF EXISTS shop;DROP DATABASE fails if someone is connected; terminate sessions with SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname = 'shop'; or, in recent versions, DROP DATABASE shop WITH (FORCE);.
Creating the database as the superuser without an owner. The superuser then owns it, and your role cannot create tables. Add OWNER, or ALTER DATABASE shop OWNER TO shop;.
Case and quoting. CREATE ROLE Shop creates shop, since unquoted identifiers fold to lowercase; CREATE ROLE \"Shop\" creates a different, case-sensitive role that you must quote forever. Use lowercase names.
Reserved words. A role or table called user or order needs quoting. Avoid them.
Passwords in scripts. Fine for disposable local roles; never reuse them anywhere that matters.
\nOne role is enough for a personal project or a prototype. It stops being enough when more than one application or person touches the database, when you separate migrations from runtime or when you give analysts read-only access. Then use several roles and group roles (roles with NOLOGIN that hold privileges, granted to login roles with GRANT group TO user). For the full model, the manual's chapters on database roles, privileges and schemas are the reference, and connecting to PostgreSQL from the command line with psql shows how to use the new role.
A file arrives that is too large to open. The spreadsheet freezes, or the notebook is killed, or the editor gives up. The reflex is to find a bigger machine or a smarter way to open it. The better move is to stop trying to open it. A large CSV is a table in a text file, and you can ask it questions without ever holding it in memory. This guide is a decision path: what to do first, how to know what is in the file, which tool to use for which question, and what each costs, with numbers measured on one 1.3 GB, 20,000,000-row file. The articles linked from each section go into detail.
\nOpening a file means loading it. A text editor builds a line index, a spreadsheet allocates a cell for every field, pandas builds a column for every column. For a file the size of your memory, each of those fails differently: slowly, with a limit message, or with a killed process. The alternative is streaming: tools that read the file once, in order, keeping a little state, and print an answer. Almost everything you want to know about a big CSV can be had that way.
\nThe steps below go from cheap to expensive. Stop at the first one that answers your question.
\nhead -n 5 orders.csv\nfile orders.csv # encoding, line endings\nls -l orders.csv # the sizeFive lines show you the header, the delimiter, the quoting, the kind of values and the line endings. head reads only the start, so it takes no time on any size of file. If the first lines look wrong (a title, a blank line, a BOM, ; as the delimiter), that is worth knowing before any tool is chosen; see the delimiter and encoding articles. The terminal preview article covers column, csvlook and friends.
How many rows? The cheap answer is wc -l, and on this file it was wrong:
| Tool | Answer | Time |
|---|---|---|
wc -l | 20,009,857 lines | 0.83 s |
DuckDB count(*) | 20,000,000 records | 0.99 s |
qsv count | 20,000,000 records | 2.20 s |
Python csv.reader | 20,000,001 (with header) | 14.74 s |
The 9,856-line difference is the number of records whose note contains a line break: a quoted field can span lines, and wc counts newlines. The counting article has the full table. The practical rule: use a parser to count records, and use wc -l only to detect that something is odd (if lines minus one differs from the record count, there are multi-line fields, and every line-based tool below needs care). An estimate from file size over average row size is nearly free, but sample from several places in the file: the first 100,000 rows overestimated the count by 4%, 20 random places by 0.26%.
Also check the last byte (tail -c 1): a file that does not end with a newline was cut off or is still being written (truncation checks, following a growing file).
The strongest tool for \"a large CSV and a question\" is a SQL engine that reads the file directly. DuckDB:
\nSELECT status, count(*) AS orders, round(sum(amount), 2) AS revenue\nFROM read_csv('orders.csv')\nGROUP BY 1 ORDER BY 1;That grouped the 20,000,000 rows in 1.23 s with a 718,120 KB peak, without creating a table or a copy. The reader is multi-threaded and converts only the columns the query uses. Any question that is a filter, an aggregate, a join with a small table or a sort of a small result fits. The DuckDB articles cover the options, including types and bad rows.
\nThe same job in pandas, loading everything:
\ndf = pd.read_csv(\"orders.csv\")\ndf.groupby(\"status\")[\"amount\"].agg([\"count\", \"sum\"])took 5.55 s and a 3,924,400 KB peak (3.39 s and 3.9 GB on a second run) just for the load, about three times the file. On a machine with 16 GB that is fine; on one with 4 GB the file would not load. It also read the five-digit zip column as int64, so 09587 became 9587. The memory article explains the sizes and how to estimate them before loading.
If you want pandas, shrink what it holds. Read only the columns you need and give them compact types:
\ndf = pd.read_csv(\"orders.csv\", usecols=[\"status\", \"amount\"],\n dtype={\"status\": \"category\", \"amount\": \"float64\"})Two columns held 0.18 GB with a 459,744 KB peak. If even that does not fit, aggregate in chunks:
\ntotal = None\nfor chunk in pd.read_csv(\"orders.csv\", usecols=[\"status\", \"amount\"], chunksize=1_000_000):\n part = chunk.groupby(\"status\")[\"amount\"].agg([\"count\", \"sum\"])\n total = part if total is None else total.add(part, fill_value=0)That took 6.21 s with a 150,740 KB peak and returned the same counts as DuckDB. It works for sums, counts, minima and maxima; a mean needs the sum and the count, and a median does not chunk. Three traps from the chunking article: types are inferred per chunk unless you pass dtype, identifiers lose leading zeros, and the pyarrow engine failed on this file because of the multi-line fields.
For exploration you want a few thousand rows, not a table. Five ways, from the sampling article:
\nshuf -n 1000 in 4.39 s using 2.7 MB, with --random-source to make it repeatable;
an awk reservoir in 5.05 s;
a seek to random byte offsets in 0.08 s, which favors rows after long lines;
DuckDB's USING SAMPLE reservoir(1000 ROWS) REPEATABLE (42) in 4.69 s;
a record-aware Python reservoir in 26.83 s.
Only the DuckDB and Python versions return whole records. In a 100,000-line sample from shuf and from the seek script, 45 to 67 lines were halves of a row with a line break, and neither sample parsed as CSV. A sample is a view of the file, and the sampling method is part of the result.
Sometimes the consumer cannot take the size: a spreadsheet, an upload form, a loader. Then split, with a tool that matches the need (splitting article):
\n| Tool | Time | Header in each part | Note |
|---|---|---|---|
split -l 1000000 | 3.64 s | no (first part only) | byte-exact; line-based |
| awk | 22.83 s | yes | byte-exact; line-based |
| qsv split | 6.79 s | yes | record-aware; re-serializes |
| Miller split | 35.48 s | yes | record-aware; re-serializes |
DuckDB FILE_SIZE_BYTES '100MB' | 4.74 s | yes | splits by size |
A cut that lands inside a quoted field breaks two parts, and a lenient reader like DuckDB reads the broken half as a row without complaint. Verify any split with a strict parser and a total record count. For Excel the target is under 1,048,576 rows including the header, though the better move is usually an extract or a summary (the Excel limits article).
\nA CSV compresses to 25 to 35% of its size. On a 130 MB export, gzip -6 gave 42.2 MB in 4.6 s, zstd -3 43.7 MB in about the time of gzip -1, and zstd -19 35.4 MB in 58.8 s. You rarely need to unzip: zcat file.gz | head, pd.read_csv(\"file.csv.gz\") and DuckDB's read_csv read compressed files directly. The cost is that a compressed stream has no random access, so compress in parts if readers want slices. Details and the truncated-archive trap are in the compression article and the gzip export article.
If the same file will be queried many times, pay the parse cost once. Convert to Parquet:
\nCOPY (SELECT * FROM read_csv('orders.csv')) TO 'orders.parquet' (FORMAT parquet, COMPRESSION zstd);The 130 MB, 2,000,000-row sample file became 34.2 MB in under a second, and a one-column maximum took 0.036 s instead of 0.203 s (why Parquet wins, the conversion). Decide the types when you convert: DuckDB read the amount as a DOUBLE, and a DECIMAL(10,2) keeps 103.10. Alternatively load into a database; PostgreSQL's COPY loaded 2,000,000 rows in 3.3 s (loading). A benchmark with the caveats spelled out compares the options.
**head, tail, wc, grep, cut, awk.** The Unix tools read a line at a time, use almost no memory and start instantly. They are right for looking, for counting lines, for filtering on a simple pattern and for cutting columns, as long as no field contains a comma or a line break that matters. Their weakness is that they do not understand CSV: awk -F, splits \"gift, wrap\" into two fields, and wc -l counts a two-line note twice. Use them to explore and to check; do not build a pipeline on them until you have confirmed the file is simple.
csvkit, qsv and Miller. These are CSV-aware command-line tools. They parse quotes properly, can select, filter, sort, join and summarize, and in the tests they were correct on the multi-line fields. qsv counted the 20,000,000 records in 2.20 s and split the file in 6.79 s; Miller was the slowest splitter at 35.48 s; csvkit's csvjson took 36 s with type inference on. Two cautions: they infer types unless told not to (Miller wrote +14192381408 as a number), and they re-serialize, so the output is the same records but not the same bytes.
DuckDB. A SQL engine in a single binary or a Python package. It reads CSV, compressed CSV and Parquet in place, runs on all cores, spills to disk and writes CSV, Parquet and JSON. It was the fastest tool for counting records (0.99 s) and for the grouped query (1.23 s), and it is the one I reach for first. Its leniency is the thing to watch: with default settings it read half of a split record as a row without a warning. Declare columns and use strict options when the file matters.
\npandas. The right tool when the next step is Python: reshaping, modeling, plotting. It is not a streaming tool, but with usecols, dtype and chunksize it stays small. The default load made a zip an int64, and the pyarrow engine failed on a file with line breaks in fields, so read the defaults before trusting them.
PostgreSQL and MySQL. Right when the data will be used by an application, needs indexes or constraints, or has to be shared. COPY loaded 2,000,000 rows in 3.3 s. Loading is a project, not an afterthought, and the loading guide lists the failure messages you will meet. For analysis of a file you will look at once, a database is more ceremony than a query engine.
Excel and Sheets. For summaries and small extracts. They are limited to 1,048,576 rows in Excel's case, and they convert values when they open a file (zip codes, long numbers, anything that looks like a date), so import with text columns and never use them to open a file you are going to write back.
\nParquet and the tools around it. Not a tool but a format, and the most effective single change for a file that is queried repeatedly. Convert once, with types decided, and every later query is cheaper. The cost is that the file is no longer something you can read in an editor.
\nA rough map, from experience with the tests here and not a law.
\nUnder about 100 MB, most things work, and the choice is about convenience. Open it in pandas or a spreadsheet if it has fewer than a million rows, and check for line breaks and encodings first. From 100 MB to a few gigabytes, loading everything into pandas starts to depend on memory: estimate it (about three times the file for the peak), type the columns, or switch to DuckDB. From a few gigabytes to a few hundred, stop thinking about loading: query in place, sample for exploration, compress for storage and convert to Parquet for repeated work. Beyond that, the file is better in a database or a columnar store, split by a meaningful key such as day, so that a query reads only the part it needs (partitioning with PARTITION_BY wrote seven country directories from the test file in 12.17 s).
None of these boundaries is sharp. A file of two long text columns is heavier than the same size of integers, and a wide file with fifty columns costs more per row than a narrow one. Let a small experiment on a slice of your data choose, as the benchmarking article describes.
\n| You want to... | Use |
|---|---|
| See what is in the file | head, file, tail -c 1 |
| Count records | DuckDB or qsv, not wc -l |
| Answer an aggregate or a filter | DuckDB over the CSV |
| Explore a few rows | a DuckDB or Python sample |
| Use pandas anyway | usecols, dtype, chunks |
| Hand it to a spreadsheet | an extract, a summary, or parts under 1,000,000 rows |
| Keep or move it | gzip or zstd |
| Query it repeatedly | Parquet, or a database |
| Watch it grow | tail -F, with care for partial lines |
Say a colleague sends orders-2025.csv, 1.3 GB, and asks how much revenue came from refunded orders in Brazil, by month. A sensible session runs like this, and the whole thing takes minutes.
First, ls -l and head -n 5: the size is 1.3 GB, the header has eight columns, created_at is an ISO timestamp, amount has two decimals, the delimiter is a comma. Second, a record count with DuckDB, SELECT count(*) FROM read_csv('orders-2025.csv'), about a second, and 20000000. That is far over a spreadsheet's limit and fine for a query engine. Third, a check that nothing is odd: the last byte is a newline (tail -c 1), wc -l is 9,856 higher than the count (so there are multi-line fields; note it), and DESCRIBE shows the types DuckDB inferred. Fourth, the question itself:
SELECT date_trunc('month', created_at) AS month, round(sum(amount), 2) AS revenue\nFROM read_csv('orders-2025.csv')\nWHERE status = 'refunded' AND country = 'BR'\nGROUP BY 1 ORDER BY 1;Twelve rows come back in about a second, and they go into a message or a spreadsheet. If the colleague then asks \"and which customers?\", the same file answers the next query in another second, because the engine streams. If the questions keep coming, convert once to Parquet and ask the next fifty against that. At no point did anything need the whole file in memory, and at no point did anyone open it.
\nThe mistakes to avoid in this session are small ones. Do not trust wc -l for the count. Do not let a tool guess that the zip is a number if you are going to join on it. Do not paste the twelve numbers without saying which file and query produced them; keep the query next to the result. And if the answer will be used for something that matters, run it a second way, with a different tool, and compare.
Before you start, decide how much you can afford. A default pandas load needs about three times the file size at peak and holds one and a half times; typed and trimmed it can hold a third of the file. DuckDB over a CSV used 718,120 KB for a grouped query on the 1.3 GB file and uses more for large joins and sorts, spilling to disk when it must. A chunked loop needs about the chunk size times the width of the columns you keep, and 150 MB was enough here. If the budget is tight, rank the tools by their peaks in this guide rather than by their speeds.
\nTime works the same way. Counting is a pass over the file, about a second per gigabyte for a good parser on a warm cache and several seconds from a cold disk. A query is one pass or a few; a conversion to Parquet is one pass plus a write. If you expect to read the file more than a handful of times, a pass per question adds up, and the conversion repays its cost. If you expect to read it once, do not convert. The loading benchmark lists what to hold constant when you compare.
\nA vendor sends a 3 GB file every Monday. Do not open it by hand. Write a short script that checks the size and last byte, counts records with DuckDB, validates the header and a few columns (validation guide), and writes the files you actually need in Parquet. Keep the original compressed. Compare the counts week over week; a file that is half the usual size is a failed export.
\nA one-off 50 GB dump on a laptop with 8 GB. Stay with streaming tools. DuckDB spills to disk for large aggregations; if it complains, reduce the question or process in key ranges. Sample first to find the columns that matter, then query only those. Do not decompress to disk if you can read the compressed file in place.
\nA file that must go into Excel. It is the summary or the extract that goes in, not the file. Decide with the recipient what they are going to do with it, produce that, and name it for what it holds (the Excel article).
\nA file that is still being written. Follow it with tail -F and act only on complete lines (following a CSV); do not count or load it until the writer is done, or take a snapshot copy first.
A file that will not parse. Check the structure before blaming the tool: delimiter, quoting, ragged rows and encoding. The counting rows and columns article and the cleaning hub cover what to do about it.
\nKeep the source immutable. Every derived file (sample, part, Parquet, extract) should be rebuildable from the original with a command you saved. Store a checksum of the original and note it in the derived file's name or manifest.
\nName derived files for what they are. orders-br-march.csv, orders-sample-1000-seed42.csv, orders.parquet made from orders-2025.csv on a given date. A sample that looks like the real thing is a hazard.
Prefer parsers to line tools when the data is unknown. Line tools are fast and correct only when no field contains a line break. Check once, and write the answer down.
\nSay what the types are. Identifiers are text, money is a decimal, dates are dates. Inference is for exploring and not for pipelines.
\nVerify in a second way. A count from wc and a count from a parser, a sum from DuckDB and a sum from pandas, a record found by grep and in the output. Agreement between two methods is cheap evidence, and disagreement is worth an hour.
Measure, then move on. Most of the time, the first streaming query answers the question. The elaborate machinery (chunks, partitions, conversions) is for the cases in which it does not, and the numbers above tell you roughly when that is.
\nFour hazards recur, and each of the spokes found at least one.
\nLine breaks inside fields. They make wc -l wrong, split -l, shuf and a seek sample unsafe, and a naive follower fragile. Find out early whether the file has any; if it does, use parser-aware tools, or have the producer remove them.
Type inference. Defaults turn identifiers into numbers, dates into strings and decimals into floats. Say what each column is.
\nSilent leniency. DuckDB read half a record as a row, pandas padded a short row, XlsxWriter dropped rows past the limit with a return code. Verify with a strict parser or a second tool.
\nThe cache. All the timings here are warm. On a cold disk or a network share, the I/O dominates and the ranking of tools compresses.
\nIt helps to name the problems, because the same few appear in every article of this cluster.
\nPeak versus resident memory. A load needs more memory while it runs than the result occupies afterwards. Plan for the peak, which is what the operating system or the scheduler enforces.
\nLine versus record. A line ends at a newline byte. A record ends at a newline that is not inside quotes. The two differ by the number of multi-line fields, and a tool that counts one when you wanted the other gives a plausible wrong answer.
\nStreaming versus materializing. A streaming tool keeps a bounded amount of state and passes over the data once. A materializing tool builds the whole table. The first scales with time, the second with memory.
\nCache. A repeated run on the same file is quicker because the operating system keeps the bytes in memory. A benchmark that does not say which it measured is half a benchmark.
\nLeniency. Many readers accept input that a strict parser rejects, by padding, skipping, truncating or guessing. Leniency is helpful for exploration and harmful for validation, and a reader that never complains is not evidence of a clean file.
\nDerived artifact. A sample, a part, a Parquet file or an extract made from the source. It carries the source's checksum or name in your notes, and it can be thrown away and rebuilt.
\nWith those six words, most conversations about \"the file is too big\" become specific enough to solve: is it the peak, the lines, the streaming, the cache, the leniency or the artifact?
\nWhatever you did, check the result against the source in a second way: a count from a different tool, a total you know, a record you can find by eye. For derived files (parts, samples, Parquet), check the row count and a checksum of the content. Keep the commands. The validation guide has the checks to run on the file before you trust it at all.
\nSometimes the size is a symptom. A file that is 20 GB because every row repeats a long text column could be a table plus a lookup. A daily full export that nobody diffs could be an incremental one. A request to \"open the whole file\" could be a request for three numbers. Ask what the reader will do with it before you pick a tool. And if the data lives in a database, query the database: it can do what these tools do without the export, and the export is where line breaks, types and encodings get damaged in the first place (exporting from databases).
\nFurther reading: DuckDB CSV import; pandas scaling to large datasets; Excel specifications and limits.
","date_published":"2026-11-11T00:22:00+00:00","tags":["CSV"]},{"id":"https://datahacks.dev/library/well-formed-valid-xml","url":"https://datahacks.dev/library/well-formed-valid-xml","title":"Well Formed Versus Valid XML and the Difference in Practice","summary":"Well formed means the syntax parses, valid means it follows a DTD or XSD. Ten broken files and two schemas run through xmllint show what each catches.","content_html":"XML has two levels of correctness, and most of the confusion about it comes from mixing them up. A document can be well formed: it follows the grammar of XML itself, so a parser can read it into a tree. And it can be valid: it also follows a set of rules for one particular kind of document, written in a DTD or an XML Schema. Every valid document is well formed, and most well-formed documents are not valid for anything, because nobody has written the rules.
\nI ran a batch of small broken files through xmllint to see what each level catches, and what neither does.
A parser has to refuse a document that breaks the grammar, because it cannot build a tree from it. The rules are short, and each one has a failure message. Ten files, each broken in one way, with the first line xmllint --noout printed:
unclosed Opening and ending tag mismatch: b line 1 and a <a><b>t</a>\ntwo-roots Extra content at the end of the document <a/><b/>\nunquoted-attr AttValue: \" or ' expected <a x=1/>\ndup-attr Attribute x redefined <a x=\"1\" x=\"2\"/>\namp xmlParseEntityRef: no name <a>Tom & Jerry</a>\nlt-in-text StartTag: invalid element name <a>1 < 2</a>\ncase Opening and ending tag mismatch: B line 1 and b <a><B>x</b></a>\nno-root Document is empty (empty file)\ntext-before Start tag expected, '<' not found text<a/>\nspace-before-decl XML declaration allowed only at the start of the document (space, then <?xml ...?>)The rules that these reflect: tags must be closed and nested properly, there is exactly one root element, attribute values are quoted, attribute names are unique within an element, the characters & and < are reserved in text, names are case sensitive, and the XML declaration, if present, is the very first thing in the file with nothing before it. Not one of these errors is about meaning. They are about whether a tree can be built.
That last rule is the one that surprises people most. A single space or a byte order mark in the wrong place breaks a file that looks fine (content is not allowed in prolog explains the usual cause).
\nProgramming-language parsers apply these rules and nothing more. Python's ElementTree and lxml both refused all ten files (ParseError and XMLSyntaxError) and both accepted the good file and the one that breaks the DTD, whatever the elements are called. Browsers and the XML parsers of other languages follow the same specification.
A well-formed catalog can still be nonsense for the system that reads it. My example, a book with no author and a price in a currency the shop does not support:
\n<catalog><book id=\"b1\"><title>T</title><price currency=\"GBP\">1</price></book></catalog>xmllint --noout printed nothing: it is well formed. The rules for a catalog live in a DTD:
<!ELEMENT book (title, author+, price, tags?)>\n<!ATTLIST price currency (USD|EUR) #REQUIRED>and checking against it:
\n$ xmllint --noout --dtdvalid catalog.dtd book.xml\nelement book: validity error : Element book content does not follow the DTD,\n expecting (title , author+ , price , tags?), got (title price)\nelement price: validity error : Value \"GBP\" for attribute currency of price is not among the enumerated set\nDocument book.xml does not validate against catalog.dtdTwo errors: the missing author, and the currency. The good catalog printed valid against DTD. The DTD is the older mechanism and has limits: it knows element order and counts and a few attribute types, but nothing about the content of price beyond \"text\".
An XML Schema (XSD) is the more expressive tool. It can say that price is a decimal and that the currency is one of two values:
$ xmllint --noout --schema catalog.xsd bad.xml\nElement 'price', attribute 'currency': [facet 'enumeration'] The value 'GBP' is not an element of the set {'USD', 'EUR'}.\nbad.xml fails to validateThe message is more specific, with the failed rule named (facet 'enumeration'), and an XSD can check types: a price of twelve fails xs:decimal, an id that repeats fails xs:ID uniqueness. Schemas are also longer to write than DTDs. For anything beyond a toy, or when a partner defines the format, the schema is usually supplied and you validate against it.
Well-formedness protects the parser. It guarantees that a tree exists. If you are only reading a file to extract a few values, it is often the only check you want.
\nValidity protects the consumer. It guarantees the structure that the code downstream assumes: the elements it reads are there, in the right number, with values it can interpret. It is a contract, and the right place to enforce a contract is at the border: when you receive a file from a partner, and when you send one.
\nNeither of them protects against wrong content that the rules allow. A valid price of 0.01 for a book that costs 12.50 passes everything.
Neither level knows your business rules. A valid catalog can list a book at a price of zero, an author that is the empty string, or two books with the same title. Those checks belong in code that runs after the parse, or in Schematron-style rules, which XML has and which few teams use. For most data feeds the pragmatic split is: the parser guarantees well-formedness, a schema (when one exists) guarantees structure, and a short function guarantees that the values make sense. Write all three as tests against a folder of sample files, valid and broken, so the day a partner changes the feed you learn which level failed (a cheat sheet of XML syntax rules lists the rules the first level enforces, and what a DTD is and why you rarely need one now covers the second).
\nIn the command line, xmllint does both:
xmllint --noout file.xml # well-formed only\nxmllint --noout --dtdvalid file.dtd file.xml # against an external DTD\nxmllint --noout --schema file.xsd file.xml # against an XSDNotice the first line: a document that declares a DOCTYPE with an external DTD was not validated by plain xmllint --noout, which only checks syntax (and my deliberately invalid example passed). Validation has to be requested. In code, lxml provides etree.DTD and etree.XMLSchema objects whose validate() returns a boolean and whose error_log explains failures (parsing and writing XML in Python covers the library). You can also look at a document's structure in the XML formatter before you decide what rules to write.
The specifications are short enough to skim: the XML 1.0 recommendation defines well-formedness and validity in its second section, and XML Schema Part 0: Primer is the gentlest introduction to XSD.
\nYou are reading a file you trust. A config file in your own repository does not need a schema. The parse error is enough.
\nThe format is loose on purpose. HTML-like documents, user-written notes and extensible vocabularies are checked by code that picks out what it understands and ignores the rest.
\nThe consumer is tolerant. If the code reads price and ignores everything it does not know, a schema adds friction without removing a failure.
You validate by other means. A test suite that parses real samples and asserts on the extracted values is a form of validation, and is often more useful than a schema nobody maintains.
","date_published":"2026-11-10T15:55:00+00:00","tags":["XML"]},{"id":"https://datahacks.dev/library/excel-xls-xlsx-xlsb","url":"https://datahacks.dev/library/excel-xls-xlsx-xlsb","title":"XLS Versus XLSX Versus XLSB and Which to Hand Over","summary":"Compare the XLS, XLSX and XLSB Excel formats on limits, size, macros and library support, detect which one a file really is, and pick one to hand over.","content_html":"Three file extensions cover almost every Excel file you will be given, and they are not interchangeable. A system that exports .xls, a colleague who saves .xlsb to make a heavy file smaller, and a script that expects .xlsx will each cause a different error, and the error messages rarely mention the format. This post compares the three on the properties that matter for moving data, shows how to tell what a file really is regardless of its name, and ends with a recommendation for what to send. It is part of the Excel cluster that starts with what is inside an Excel file.
XLS is the Excel 97 to 2003 format, a binary container (OLE2, also called Compound File Binary) holding a stream of records. It has 65,536 rows and 256 columns per sheet, and its limits on styles and colors are far lower than modern Excel. It cannot be read by unzipping. Many old systems, report generators and instruments still write it because the library they use was written in 1999.
\nXLSX is the Office Open XML format introduced with Excel 2007: a zip of XML parts, 1,048,576 rows and 16,384 columns, no macros. It is an open standard (ECMA-376 and ISO/IEC 29500), and you can read it with an archive tool and a text editor. It is the format almost every library targets first.
\nXLSB is the Excel binary workbook: also a zip package, but the parts are binary records and not XML. It stores the same data as XLSX with the same limits, usually in a smaller file that Excel loads faster, since parsing binary records is cheaper than parsing XML. It can hold macros. Its drawback is support: Excel and a few libraries read it, and many tools do not.
\nThere is a fourth member of the family worth naming. XLSM is the XLSX layout with macros enabled. It is XML, a zip, and as readable as XLSX, but a vbaProject.bin part carries the code, and many mail filters block it.
| Property | XLS | XLSX | XLSB |
|---|---|---|---|
| Container | OLE2 binary | Zip of XML | Zip of binary parts |
| Rows by columns | 65,536 by 256 | 1,048,576 by 16,384 | 1,048,576 by 16,384 |
| Macros | Yes | No (use XLSM) | Yes |
| Typical size | Larger than XLSX for the same data | Baseline | Often the smallest |
| Readable without Excel | Hard | Easy | Hard |
| Library support | xlrd, calamine | Nearly everything | pyxlsb, calamine |
Support differs more than the table suggests, and this is where handovers fail. In Python:
\nopenpyxl reads and writes XLSX and XLSM. It does not read XLS or XLSB, and raises an error for them.
xlrd reads XLS. Version 2.0 and later dropped XLSX support, so it is only for the old format.
pyxlsb reads XLSB, with fewer features than openpyxl has for XLSX.
python-calamine is a Rust-based reader that opens XLS, XLSX, XLSB and ODS through one interface. pandas can use it as engine=\"calamine\" in recent versions (choosing between openpyxl, xlsxwriter and calamine).
pandas picks an engine from the extension when it can, and fails with an instruction to install one when it cannot (fixing the missing optional dependency openpyxl error).
DuckDB's Excel extension reads XLSX. The database import wizards of SQL Server and MySQL Workbench have their own rules, and some need a driver for the format: SQL Server's tooling relies on the Access Database Engine (ACE) provider for XLSX and XLSB.
\nIn practice, if your pipeline is written with openpyxl and someone sends an .xlsb, the pipeline stops. Converting the file once in Excel (Save As, XLSX) is the fastest fix, and the permanent fix is to agree on a handover format.
Files get misnamed constantly. A web application \"exports to Excel\" by returning an HTML table with an .xls extension, because Excel opens it. Another writes a tab-separated text file and calls it .xls. A script renames a CSV. Excel opens all of these, with a warning that the file format and extension do not match, and every library fails differently. Check the first bytes of the file:
import sys, zipfile\n\nSIGNATURES = {\n b\"\\xd0\\xcf\\x11\\xe0\\xa1\\xb1\\x1a\\xe1\": \"OLE2 binary (real XLS, or an old DOC or PPT)\",\n b\"PK\\x03\\x04\": \"zip package (XLSX, XLSM, XLSB, ODS or another zip)\",\n}\n\npath = sys.argv[1]\nwith open(path, \"rb\") as f:\n head = f.read(8)\n\nkind = next((v for k, v in SIGNATURES.items() if head.startswith(k)), None)\nif kind is None:\n text = head.lstrip(b\"\\xef\\xbb\\xbf \\r\\n\\t\")[:1]\n kind = \"HTML or XML text\" if text == b\"<\" else \"plain text (CSV or TSV?)\"\n\nprint(path, \"->\", kind)\n\nif head.startswith(b\"PK\"):\n names = zipfile.ZipFile(path).namelist()\n if \"xl/workbook.bin\" in names:\n print(\" workbook.bin present: XLSB\")\n elif \"xl/workbook.xml\" in names:\n macros = \"xl/vbaProject.bin\" in names\n print(\" workbook.xml present: XLSX\" + (\" with macros (XLSM)\" if macros else \"\"))The eight-byte OLE2 signature D0 CF 11 E0 A1 B1 1A E1 marks a binary XLS. Every zip begins with PK. Inside a zip, xl/workbook.xml marks the XML flavor and xl/workbook.bin marks XLSB. A file that starts with < is HTML or XML, and one that starts with something else printable is probably delimited text. For a web \"export\" renamed to .xls, pandas can read it with read_html, not read_excel, and the quicker fix is to ask the source for a CSV.
The choice depends on who reads the file.
\nA person with Excel. Send XLSX. Every version since 2007 opens it, Google Sheets and LibreOffice open it, and macros are not an issue. Choose XLSB only for an internal file that is slow to open, where everyone who touches it uses desktop Excel, and keep an XLSX copy for anyone else.
\nA program written in any language. Send CSV if the data is one table, and see Excel versus CSV for sharing data with other systems. If styles, several sheets or formulas matter, send XLSX with the data on a single sheet, a header in row 1 and identifiers stored as text.
\nAn archive. XLSX, with the data also exported as CSV. The XML layout is documented and readable in the long term, which cannot be said for the binary formats.
\nA legacy system that demands XLS. Send XLS, keep it under 65,536 rows, and make sure no more than 256 columns exist. Write it with a library, not by renaming a CSV, and test the file in the receiving system. Treat it as a bridge that you plan to retire.
\nNever hand over XLSM unless the receiver asked for the macros. A file with macros is blocked by many filters and treated with suspicion by good habit.
\nExcel's Save As converts one to another and is the most faithful option for a single file. For many files, LibreOffice can run without a window:
\nsoffice --headless --convert-to xlsx --outdir converted old/*.xlsThe command reads every .xls file in old and writes a .xlsx file for each into converted. Conversion is not lossless. Formatting that has no equivalent is approximated, charts can change, and formulas are recalculated by the converting program, which may differ from Excel in edge cases. Compare totals before and after, as in validating row counts after an Excel import. With python-calamine you can skip the conversion and read the old file directly, which avoids the question for data-only use.
Be careful about the direction. Converting XLSX to XLS to satisfy an old system silently drops every row after 65,536 in some tools, and fails in others. Check the row count first.
\nXLSB is often described as 50 to 70 percent smaller than XLSX and several times faster to open. Claims like these come from vendor guidance and depend heavily on the content: a workbook that is mostly text compresses well in either, and one with large formula chains gains the most from the binary layout. I have not measured them here. If you are considering XLSB for speed, convert a representative file and time the open in Excel and the load in your pipeline. XLSX already compresses with zip, so the gap is smaller than the gap between XLS and XLSX was.
\nThe reverse matters more for data engineering: XLSX parsing is slow because XML parsing is slow, and a million-row file takes noticeable time in any library. A fast reader helps (reading Excel files faster with python calamine), and so does reading in a mode that streams instead of loading everything (reading large Excel files in read-only mode).
\nThe data is more than a million rows. All three formats stop at 1,048,576 rows. Use CSV, a database or a columnar file such as Parquet (Excel row and column limits).
\nThe receiver is a machine and the data has types. A spreadsheet carries only numbers, strings, booleans and dates-as-numbers. Typed formats such as Parquet or Avro, or JSON with a schema, say more.
\nYou need version control. Binary and zipped formats diff poorly. Keep the source as CSV in the repository and generate workbooks from it.
\nThe file is a template with logic. Formulas and macros that the receiver must keep working are a reason for XLSX or XLSM, but also a reason to ask whether the logic belongs in code.
\nThe structure of the package formats is described in ECMA-376, and the binary workbook format in Microsoft's MS-XLSB specification. For a quick look at a small table as JSON, use the CSV to JSON converter.
","date_published":"2026-11-10T00:34:00+00:00","tags":["Excel"]},{"id":"https://datahacks.dev/library/reading-yaml-front-matter-markdown","url":"https://datahacks.dev/library/reading-yaml-front-matter-markdown","title":"Reading YAML Front Matter From Markdown Files in Python","summary":"Split a Markdown file into YAML metadata and body with one regex, then handle dates, CRLF, a BOM, a missing closing line and body text with dashes.","content_html":"Static site generators, documentation tools and note-taking apps keep a bit of YAML at the top of each Markdown file: a title, a date, tags, a draft flag. This is called front matter. If you write a script that builds an index, finds drafts or renames files from their metadata, you need to split the file and parse the top block. The regular expression takes one line and the edge cases take the rest of this post, with each tested.
\n---\ntitle: Hello\ndate: 2026-10-27\ntags: [a, b]\ndraft: no\n---\n# Body\nSome textThe first line must be exactly three dashes. The block ends at the next line of three dashes (some tools also accept three dots). Everything after that line is the body. Hugo, Jekyll, Eleventy and many others use this convention, and the loader in your script needs to follow the same rules.
\nimport re\nimport yaml\n\nFM = re.compile(r\"\\A?---[ \\t]*\\r?\\n(.*?)\\r?\\n(?:---|\\.\\.\\.)[ \\t]*(?:\\r?\\n|\\Z)(.*)\\Z\", re.S)\n\ndef split(text):\n m = FM.match(text)\n if not m:\n return None, text\n return yaml.safe_load(m.group(1)), m.group(2)The pattern is anchored at the start (\\A), allows an optional byte order mark, requires --- on the first line, captures lazily (.*?) up to the first closing line and captures the rest as the body. re.S lets the dot match newlines. The lazy match is what protects the body: it stops at the first closing marker. Results on a normal file:
normal meta={'title': 'Hello', 'date': datetime.date(2026, 10, 27), 'tags': ['a', 'b'], 'draft': False} body='# Body\\nSome text\\n-...'\n{'title': 'str', 'date': 'date', 'tags': 'list', 'draft': 'bool'}The body starts at # Body and keeps the later --- line, which in Markdown is a horizontal rule, untouched. The metadata types show two YAML conversions to be aware of. date: 2026-10-27 is a datetime.date, because YAML resolves the timestamp. And draft: no is the boolean False, because PyYAML implements YAML 1.1 (the Norway problem and YAML boolean surprises). That is convenient for a draft flag and a trap for any other field that happens to hold no, on or yes as text.
If you pass the metadata to json.dumps to build an index, the date breaks it:
json.dumps: Object of type date is not JSON serializableThree fixes. Convert in the dump: json.dumps(meta, default=str) writes \"2026-10-27\". Or quote the date in the front matter so it is a string from the start. Or convert deliberately after loading: meta[\"date\"] = meta[\"date\"].isoformat() when it is a date. The last is the most explicit and lets you handle both date and datetime. The first is the least code and also converts other objects you did not expect to str, so check what you get.
A related trap: a date written with a time zone offset, a time of day without a date, or a malformed date such as 2026-13-45 is a string or an error depending on the form. If your site depends on dates, validate them after loading (validating YAML configuration with pydantic works on front matter dictionaries too).
crlf meta={'title': 'Win'} body='Body\\r\\n'\nbom meta={'title': 'Bom'} body='Body\\n'\nno front matter meta=None body='# Just a heading\\nt...'\nempty front matter meta=None body='---\\n---\\nBody\\n'\nnot closed meta=None body='---\\ntitle: Open\\nBo...'\nbad yaml YAMLError mapping values are not allowed here (line 1)Windows line endings work because the pattern accepts \\r?\\n. The body keeps its \\r\\n endings, so normalize them if your output tool cares.
A byte order mark at the start (some Windows editors add one) is allowed by ?. Without it the first line is not --- and the split returns no metadata, a bug that appears only for files from one editor. Open files with encoding=\"utf-8-sig\" to strip the mark at the source.
No front matter returns (None, text). Callers must handle None, for instance with meta or {}.
An empty block (two dash lines in a row) did not match the pattern in this version, because the pattern requires a line break before the closing marker, and the result was no metadata with the body unsplit. If empty blocks occur in your files, adjust the pattern to make the content group optional, and decide whether an empty block means an empty dictionary.
\nAn unclosed block returns no metadata, and the whole file, including the stray ---, is the body. A script that silently treats such a file as having no front matter will hide a typo. Consider raising when the first line is --- but no closing line exists.
Invalid YAML in the block raises a normal YAMLError. Its line number counts from the start of the block, not the file, so line 1 of the error is line 2 of the file. Add the offset (one line for the opening marker, or two with a BOM) before showing it to a person.
You do not have to write the splitter. The python-frontmatter package parses front matter in YAML, TOML or JSON and returns a post object with metadata and content, and it handles several of the cases above. yq can read front matter on the command line: yq --front-matter=extract '.title' post.md printed the metadata of a Markdown file in a shell script. Static site generators expose it to templates already. Write your own when you need to control the exact rules, as for a linter that enforces required fields.
A checker for a content repository is a useful application: required keys present, dates valid, tags a list, no drafts in the publish directory. Walk the files with glob, split each, validate the dictionary and print one line per problem with the file name. Run it in CI. Because front matter is user-written YAML, everything in this cluster applies: quote titles that contain a colon, quote anything that could be misread, watch for duplicate keys and keep the block short (YAML strings, quoting and when you need quotes). A title such as Fix: the parser is the commonest failure, and it produces \"mapping values are not allowed here\" on line 1 of the block.
The cluster hub is reading and writing YAML in Python safely. The loading call is explained in loading YAML in Python with PyYAML safe_load. The convention is described in the Jekyll front matter documentation and the library at python-frontmatter. Check a block's types in the YAML to JSON tool.
\nThe files use several metadata formats. TOML between plus signs and JSON between braces are also used. A library handles all three.
\nYou rewrite the front matter. Editing and writing it back needs a round-trip YAML library to keep comments and order (preserving comments and order with ruamel yaml).
\nThe body contains lines that look like the closing marker. In a fenced code block, a line of three dashes could be mistaken only if it comes before the real closing line, which it cannot. The lazy match is safe for well-formed files.
\nYou need exact positions. For error reporting inside editors, track the offset of the block, not just its content.
","date_published":"2026-11-09T15:11:00+00:00","tags":["YAML"]},{"id":"https://datahacks.dev/library/installing-mysql-8-4-macos","url":"https://datahacks.dev/library/installing-mysql-8-4-macos","title":"Installing MySQL 8.4 on macOS Linux and Windows Step by Step","summary":"Install MySQL 8.4 LTS on macOS, Ubuntu and Windows: package sources, first start, the root account, secure setup and checks that the server works.","content_html":"Installing MySQL looks routine and has a few choices that matter later: which release line, which package source, where the data lives and how the root account starts out. This article installs MySQL 8.4 LTS on the three common desktop and server platforms, walks through the first start and secures the result. It sticks to supported installers from the project and Homebrew, and it ends with checks that the server is what you expect before an application connects.
\nMySQL has long term support releases and innovation releases. Version 8.4 is the LTS line, supported for years with bug and security fixes and no feature changes, and it is the right default for a database that will run for a long time. Innovation releases such as 9.x ship new features quickly and are supported only until the next one. Version 8.0 is at or near end of life according to Oracle's lifetime policy, so do not start new work on it. The reasoning is in which MySQL version to choose for a new project. MariaDB is a separate fork with its own packages and differences, not covered here.
\nbrew install mysql@8.4\nbrew services start mysql@8.4Homebrew installs a versioned formula and puts binaries under its prefix without linking them to your path, so add them: echo 'export PATH=\"/opt/homebrew/opt/mysql@8.4/bin:$PATH\"' >> ~/.zshrc on Apple silicon (the prefix is /usr/local on Intel Macs). The data directory is /opt/homebrew/var/mysql and the configuration file /opt/homebrew/etc/my.cnf. The root account starts with no password, reachable only from localhost. The installer from the MySQL download page is an alternative that registers a system service and prompts for a root password.
The distribution repositories often carry an older version, so add the MySQL APT repository. Download the mysql-apt-config package from the MySQL downloads page, install it with sudo dpkg -i, choose the 8.4 LTS server in its menu, and then:
sudo apt update\nsudo apt install mysql-server\nsudo systemctl status mysqlThe installer asks for a root password, or sets up authentication through the socket so that sudo mysql works without one. Data lives in /var/lib/mysql, configuration in /etc/mysql/, and logs in /var/log/mysql/ or the journal. Red Hat family systems use the MySQL Yum repository with dnf install mysql-community-server; the first start writes a temporary root password to the error log (sudo grep 'temporary password' /var/log/mysqld.log) that you must change at first login.
Download the MySQL Installer (MSI) from the MySQL website, choose the Server only or Developer setup, pick 8.4, and set the root password and the service name when the wizard asks. The server runs as a Windows service, MySQL84 by default, and its files sit under C:\\\\ProgramData\\\\MySQL\\\\MySQL Server 8.4\\\\ with my.ini as the configuration file. Use the \"MySQL 8.4 Command Line Client\" shortcut or add the bin directory to the path. Keep the data directory off folders that a backup or antivirus tool scans aggressively, since it hurts performance.
On first start the server initializes the data directory. Depending on the package, root has an empty password, a generated temporary one or socket authentication. Replace it at once:
\nALTER USER 'root'@'localhost' IDENTIFIED BY 'a-long-generated-password';Version 8.4 defaults to the caching_sha2_password plugin, and the older mysql_native_password is disabled by default, which matters for old clients and is covered in fixing authentication plugin caching_sha2_password errors. Then run the hardening script:
sudo mysql_secure_installationIt sets the password policy, removes anonymous accounts and the test database, and disallows remote root login. The details of what it changes are in removing anonymous users and test databases.
Connect and ask the server:
\nSELECT VERSION();\nSHOW VARIABLES LIKE 'character_set_server';\nSHOW VARIABLES LIKE 'datadir';\nSHOW VARIABLES LIKE 'port';You should see a version starting with 8.4, utf8mb4 as the server character set (the right default; see why utf8mb4 is the only charset you should use), the data directory you expect and port 3306. Check that the server listens only where intended: SHOW VARIABLES LIKE 'bind_address' is 127.0.0.1 or * depending on the package, and opening it to the network without a firewall and TLS is a decision to make later with care. Look at the error log once to confirm a clean start with no warnings about deprecated settings.
Write down how the server was installed and how it will be updated. Package-managed installs receive minor updates through the package manager, and a monthly check of the release notes for security fixes is a good habit. Homebrew upgrades replace the formula in place, and a major version change needs a deliberate step. Pin the version in configuration management so that a new machine matches the old one. Keep the configuration file in version control, and note any non-default setting with the reason, since the file is the part of the install that people forget to document.
\nIf the service does not start, read the error log, whose location is SHOW VARIABLES LIKE 'log_error' or the journal. Typical causes are a port already in use by another MySQL or MariaDB, permissions on the data directory, or a leftover data directory from another version that the new server refuses to open. ERROR 1045 Access denied for root is covered in fixing access denied for user root at localhost, and a missing socket in fixing can not connect to local MySQL server through socket. On macOS, an older Homebrew MySQL or the Oracle installer may already occupy the port; stop it first.
For development, a container keeps the server, its version and its files apart from your system and removes the platform differences above. One command starts a throwaway or persistent server, as shown in running MySQL in Docker with a data volume. Use native packages for a server that stays up, a machine where Docker is not available or when you want the service manager integration. For production on a managed cloud, none of this applies, since the provider installs and patches the server.
\nInstall 8.4 on the platform you use, change the root password, run the secure script and create an application user. Save the commands you used in your notes so the next machine is a copy. The manual documents every platform in installing MySQL, and the Homebrew formula is described at formulae.brew.sh. The hub is getting a MySQL server running and usable.
","date_published":"2026-11-07T00:35:00+00:00","tags":["MySQL"]},{"id":"https://datahacks.dev/library/json-data-types-missing","url":"https://datahacks.dev/library/json-data-types-missing","title":"JSON Data Types and What Is Missing From Them Entirely","summary":"JSON has six value types and no date, binary, integer, decimal, set or undefined. See what Python and JavaScript do to each missing type when they serialize.","content_html":"JSON looks like it has types because JavaScript does. It has six, and they are fewer than the ones you use. A value is an object, an array, a string, a number, true or false, or null. There is no date, no binary blob, no set, no map with non-string keys, no decimal that stays exact, no distinction between an integer and a float, no undefined and no way to write infinity. Whatever your program has beyond those six, it has to squeeze into them when it writes JSON and rebuild when it reads, and the squeeze is where data changes without an error. This article runs the missing types through Python and JavaScript, shows what each does and gives the encodings that keep the data intact. It builds on the grammar described in what JSON allows and what parsers quietly accept.
An object maps string names to values. An array is an ordered list. A string is Unicode text. A number is a decimal literal with no stated range. true and false are the booleans, and null is the one value for nothing. That is the full list.
Two properties follow that matter later. Object keys are always strings. And a number is just digits: JSON does not say that 1 is an integer and 1.0 a float, or that a number fits in 64 bits. Reading those distinctions is up to the parser, and different parsers read them differently (how JSON handles big numbers).
Python's json.dumps knows the six types plus a few obvious cousins: dict, list and tuple (written as an array), str, int, float, bool and None. For anything else it raises:
datetime TypeError: Object of type datetime is not JSON serializable\nset TypeError: Object of type set is not JSON serializable\nbytes TypeError: Object of type bytes is not JSON serializable\nDecimal TypeError: Object of type Decimal is not JSON serializable\ntuple [1, 2]\nint key {\"1\": \"a\"}\ntuple key TypeError: keys must be str, int, float, bool or None, not tuple\nnan NaNThe errors are the safe behavior: the library will not guess how to write a date, so you must say. The silent conversions are the dangerous ones. A tuple becomes an array, and when it is read back it is a list, so (1, 2) does not survive a round trip. An integer key becomes a string: {1: \"a\"} is written {\"1\": \"a\"}, and reading it back gives {'1': 'a'}, so the key's type changed, which is a classic cause of a lookup that works in memory and fails after a cache. And float(\"nan\") is written as the bare word NaN, which is not valid JSON; Python allows it by default, and most other parsers reject it. Turn that off with allow_nan=False, which raises ValueError: Out of range float values are not JSON compliant: nan (is NaN or Infinity valid in JSON).
To serialize a type that Python does not know, pass a default function that converts it, for example turning a datetime into an ISO 8601 string and a Decimal into a string, and raising TypeError for anything unknown so that the safe behavior remains.
JSON.stringify is more permissive, and its choices are worth knowing because they lose information without a message. The same sort of values:
undefined {\"b\":1} (the key with undefined is dropped)\nfunction {\"b\":1} (so is a method)\narray undef [null,null] (in an array, undefined and functions become null)\nDate {\"d\":\"2026-10-02T08:15:00.000Z\"}\nNaN {\"n\":null,\"i\":null} (NaN and Infinity become null)\nMap {}\nSet {}\nBigInt TypeError: Do not know how to serialize a BigInt\nSymbol key {\"y\":2}A property whose value is undefined disappears: { a: undefined, b: 1 } serializes as {\"b\":1}, so \"the field is missing\" and \"the field was set to undefined\" are the same on the wire. The same value in an array becomes null, so an array's length is preserved and its content is not. NaN and Infinity become null, which is valid JSON and a silent change of meaning: a missing measurement and a division by zero look identical. A Map and a Set serialize as {}, an empty object, with no error, which is the worst of these, because the data is gone and the output looks fine. And a BigInt raises, which is again the safe behavior. A Date has a toJSON method and becomes an ISO 8601 string in UTC with milliseconds: 2026-10-02T08:15:00.000Z. When it is read back, it is a string, not a date, and you must convert.
Each missing type needs an agreed encoding, and the right one depends on how the data is used. The usual choices:
\n| Type | Common encoding | Notes |
|---|---|---|
| Date or timestamp | ISO 8601 string, UTC or with an offset | Never a locale string; see JSON dates |
| Binary | base64 string | Adds a third in size; say the variant (standard or URL-safe) |
| Decimal and money | string, or integer minor units | Floats cannot hold 0.1 exactly |
| Big integer ID | string | Above 2^53 a float loses digits |
| Set | array, sorted if order should not matter | Document that it is a set |
| Map with non-string keys | array of [key, value] pairs, or an array of objects | Keys must be strings in an object |
| Integer vs float | the number as written, and a schema that says which | 1 and 1.0 may read the same in JavaScript |
| Missing value | omit the field, or null | Choose one; say whether they differ |
| Infinity, NaN | null, or a string, or omit | Not representable as numbers |
| Enum | string | Stable names, not numbers |
Write the choices in the interface description, ideally with a JSON Schema, and test them in both directions: write, read back, compare the typed values, not the text. A date serialized by one service as 2026-10-02T08:15:00Z and by another as 2026-10-02T08:15:00.000+00:00 is the same instant and different text, so consumers must parse, not compare strings.
JSON's null raises a design question that the type list hides. A field can be present with a value, present as null, or absent, and those are three states that many APIs collapse into two. In JavaScript, absent and undefined coincide on the wire. In databases, null means unknown. In PATCH-style updates, an absent field means \"leave as is\" and null means \"clear it\", and a client that drops null fields as an optimization breaks the second. Decide the semantics per API, write them down and test the three cases. In CSV the same question appears as an empty cell, and round tripping exports back into imports shows what happens when the distinction is lost.
If the data is full of types JSON lacks, dates, decimals, binary, sets, the encodings above work and cost you a schema and discipline. For some jobs a different format is better. Protocol Buffers, Avro and Parquet carry types in the schema; MessagePack and CBOR add binary and a few types while keeping the JSON data model; BSON adds dates, binary and 64-bit integers for MongoDB. If humans do not need to read the bytes, those formats remove the guesswork. If they do, JSON with a schema is a fair middle path.
\nWhichever way you go, make the encoding explicit in code, with a small serializer function in one place that every write goes through, and a matching deserializer for reads. The scattered str(value) calls are where a date gets written two ways. The Python behavior is documented in the json module reference, and the JavaScript rules in the MDN page for JSON.stringify.
A container is the quickest way to get a clean PostgreSQL on any machine, and it has one trap that catches nearly everyone once: containers are disposable, and a database inside a container's own filesystem is disposable with it. Remove the container and the data goes too. The fix is a volume, and with it the container becomes the right tool: start a server in a second, throw it away, start another against the same data. This article shows the command, the settings that apply only on first start, the Compose version and the mistakes around ports and versions. It is the container route in setting up PostgreSQL for development without fighting it, and the native route is in installing PostgreSQL on macOS, Linux and Windows. Once it runs, creating your first database user and schema in PostgreSQL is the next step.
\ndocker volume create pg17data\n\ndocker run --name pg17 -d \\\n -e POSTGRES_PASSWORD=dev-only-password \\\n -e POSTGRES_USER=shop \\\n -e POSTGRES_DB=shop \\\n -p 127.0.0.1:5432:5432 \\\n -v pg17data:/var/lib/postgresql/data \\\n postgres:17Each part has a job.
\n--name pg17 gives the container a stable name for later commands (docker stop pg17, docker logs pg17). -d runs it in the background.
-e POSTGRES_PASSWORD=... is required; the image refuses to start without a password for the superuser (or an explicit decision to allow trust authentication, which you should not make). POSTGRES_USER names the superuser, defaulting to postgres; POSTGRES_DB names a database to create, defaulting to the user's name.
-p 127.0.0.1:5432:5432 publishes container port 5432 on the host's loopback interface only. Written as -p 5432:5432, Docker publishes on all interfaces, which on a laptop on a shared network exposes the database. If port 5432 is taken on the host by another PostgreSQL, map a different host port: -p 127.0.0.1:5433:5432, and connect to 5433.
-v pg17data:/var/lib/postgresql/data mounts the named volume at the data directory, so the database files live in the volume.
postgres:17 pins the major version. A bare postgres means latest, which moves to the next major when it is released; an existing volume created by 17 will not start under 18 (the server says the files are from an incompatible version) and you are left with an upgrade you did not plan.
Check it:
\ndocker logs pg17 | tail -5\ndocker exec pg17 pg_isready -U shop -d shop\npsql \"postgresql://shop:dev-only-password@127.0.0.1:5432/shop\" -c \"SELECT version();\"The log should end with database system is ready to accept connections.
The image's entrypoint looks at the data directory when the container starts. If it is empty, the entrypoint runs initdb, creates the superuser and database from the environment variables and runs any initialization scripts. If it already contains a cluster, it skips all of that and starts the server.
So changing POSTGRES_PASSWORD in a command or a Compose file and restarting does nothing to an existing volume: the old password still applies. That is the single most common Docker PostgreSQL confusion (\"I set the password but it says authentication failed\"). To change a password, change it inside the database (ALTER ROLE shop PASSWORD '...'). To start over with new settings, remove the volume, which deletes the data:
docker rm -f pg17\ndocker volume rm pg17dataDo this only when you mean it. docker volume ls shows which volumes exist, and docker volume inspect pg17data shows where the data lives on the host.
Files placed in /docker-entrypoint-initdb.d/ run once, on first start, in alphabetical order. They can be .sql, .sql.gz or .sh. This is the right place for schema and seed data.
docker run --name pg17 -d \\\n -e POSTGRES_PASSWORD=dev-only-password \\\n -v pg17data:/var/lib/postgresql/data \\\n -v \"$PWD/db/init\":/docker-entrypoint-initdb.d:ro \\\n postgres:17Scripts run as the superuser against the POSTGRES_DB database. They run only when the data directory is empty, so editing them later has no effect on an existing volume; apply changes with migrations instead. A failing script aborts the initialization, and the container exits. Check docker logs for the line with the error.
Compose records the same settings in a file the whole team shares.
\nservices:\n db:\n image: postgres:17\n container_name: shop-db\n restart: unless-stopped\n environment:\n POSTGRES_USER: shop\n POSTGRES_PASSWORD: dev-only-password\n POSTGRES_DB: shop\n ports:\n - \"127.0.0.1:5432:5432\"\n volumes:\n - pgdata:/var/lib/postgresql/data\n - ./db/init:/docker-entrypoint-initdb.d:ro\n healthcheck:\n test: [\"CMD-SHELL\", \"pg_isready -U shop -d shop\"]\n interval: 5s\n timeout: 3s\n retries: 10\n\nvolumes:\n pgdata:docker compose up -d starts it, docker compose logs -f db follows the log, docker compose down stops and removes the container but keeps the volume, and docker compose down -v also removes the volumes, which deletes the data. The health check lets other services use depends_on with condition: service_healthy to wait until the database accepts connections. Keep real passwords out of the file: use an .env file that Compose reads (and that is not committed), or Docker secrets.
Other containers on the same Compose network reach the database at the hostname db (the service name) on port 5432, not localhost and not the published host port. A web service whose connection string says localhost inside its own container is talking to itself.
A named volume is managed by Docker, with good performance on all platforms and a layout you do not touch. It is the right default. A bind mount (-v ./pgdata:/var/lib/postgresql/data) maps a host directory, which makes the files visible and easy to back up, and brings trouble: on Docker Desktop for Mac and Windows the performance can be poor, permissions between host and container must line up (the data directory must be owned by the container's postgres user and mode 0700) and a stray git add can try to commit a database. Use a named volume unless you have a reason.
The mount path above, /var/lib/postgresql/data, was the image's data directory for many versions. Newer major versions of the official image moved the default data directory to a version-specific location under /var/lib/postgresql, and recommend mounting the volume at that parent path. Mounting at the old path with a new image can create an anonymous volume that you do not expect. Read the image's Docker Hub page for the exact guidance for your major version before you choose the mount, and verify with docker exec pg17 psql -U shop -c \"SHOW data_directory;\" that the path reported is the one you mounted.
A volume is not a backup. Take dumps from the running container:
\ndocker exec pg17 pg_dump -U shop -Fc shop > shop.dump\ncat shop.dump | docker exec -i pg17 pg_restore -U shop -d shop_copy --no-ownerCreate the target database first. The dump format is portable across hosts and across minor versions. To copy a volume's raw files to another machine, stop the container first; copying a running data directory yields an inconsistent copy.
\nPass server settings on the command line after the image name: postgres:17 -c log_min_duration_statement=200 -c shared_buffers=256MB. Or mount a config file and point to it with -c config_file=/etc/postgresql/postgresql.conf. Extensions that ship with the server (pg_stat_statements, pgcrypto, pg_trgm) are available in the standard image; others need a custom image built FROM postgres:17 with the package added. Extensions are enabled with CREATE EXTENSION in each database that uses them.
Give the container a reasonable shared memory size if you raise shared_buffers a lot (--shm-size=256m), since the container's default /dev/shm is small.
Authentication failed after changing the password variable. The volume already existed; see above.
\nContainer exits immediately. docker logs shows why: a missing POSTGRES_PASSWORD, an incompatible data directory or a failing init script.
Port already allocated. Another process holds 5432. Change the host port.
\nConnection refused from the host right after start. The server is still initializing; wait for the ready message or use the health check.
\nData gone. The container ran without a volume, or docker compose down -v ran. A dump is the only protection.
Permission denied on a bind mount. The directory's owner or mode is wrong for the container user.
\nDocker is a good fit for development and tests. In production it is a decision with trade-offs: you take on the care of volumes, upgrades, backups and storage performance, which a managed service or a well-run native install handles for you. On a machine with limited resources, or on Docker Desktop with slow file sharing, a native install may feel faster. And when a team shares one long-lived development database, a server with proper backups beats everyone's laptop containers. Use containers for the disposable parts. The official image page documents every variable and path, and the PostgreSQL manual on server setup covers the settings inside.
","date_published":"2026-11-06T00:23:00+00:00","tags":["PostgreSQL"]},{"id":"https://datahacks.dev/library/converting-csv-parquet","url":"https://datahacks.dev/library/converting-csv-parquet","title":"Converting a CSV to Parquet and Why You Might Want To","summary":"One DuckDB statement turned a 130 MB CSV into a 34 MB Parquet file; typed columns, dictionary encoding and min/max statistics explain the faster queries.","content_html":"CSV is a good format for sending a table to a person or another system. It is a bad format for asking the same table a hundred questions. Every query has to read all the text, split it at commas, find the quotes and convert the digits to numbers, and it does that again for each query. Parquet is the format most analytical tools use for the second job. This article converts one CSV into Parquet in one statement and then opens the file up to show what changed: the types, the layout, the compression and the statistics. Those four things are the entire reason for the speed-up, and understanding them tells you when the conversion is worth doing.
\nWith DuckDB, reading a CSV and writing Parquet is one COPY statement (the Parquet section of the DuckDB documentation lists the options):
head -n 3 work/orders-2m.csv\nduckdb -box -c \"DESCRIBE SELECT * FROM read_csv('work/orders-2m.csv')\"order_id,customer_id,zip,amount,created_at,status,country,note\n1,22313,99154,103.1,2025-11-22 13:08:24,shipped,GB,rush order\n2,193489,34494,31.59,2025-03-03 08:03:33,failed,BR,leave at door\n┌─────────────┬─────────────┬──────┬──────┬─────────┬───────┐\n│ column_name │ column_type │ null │ key │ default │ extra │\n├─────────────┼─────────────┼──────┼──────┼─────────┼───────┤\n│ order_id │ BIGINT │ YES │ NULL │ NULL │ NULL │\n│ customer_id │ BIGINT │ YES │ NULL │ NULL │ NULL │\n│ zip │ VARCHAR │ YES │ NULL │ NULL │ NULL │\n│ amount │ DOUBLE │ YES │ NULL │ NULL │ NULL │\n│ created_at │ TIMESTAMP │ YES │ NULL │ NULL │ NULL │\n│ status │ VARCHAR │ YES │ NULL │ NULL │ NULL │\n│ country │ VARCHAR │ YES │ NULL │ NULL │ NULL │\n│ note │ VARCHAR │ YES │ NULL │ NULL │ NULL │\n└─────────────┴─────────────┴──────┴──────┴─────────┴───────┘The first part of the output is what DuckDB inferred from the CSV before writing anything. order_id and customer_id are BIGINT, created_at is TIMESTAMP, zip, status, country and note are VARCHAR, and **amount is DOUBLE**. Two of those are decisions you may not agree with. zip stayed text, rightly, because DuckDB saw leading zeros (00500). And amount became a floating-point number, though it holds money. A CSV carries no types, so the converter chooses them, and Parquet records the choice permanently. I come back to this below.
Writing with different compression codecs:
\nfor c in snappy zstd gzip uncompressed; do\n printf '%-13s' $c\n s=$(date +%s.%N)\n duckdb -c \"COPY (SELECT * FROM read_csv('work/orders-2m.csv')) TO 'work/pq/o_$c.parquet' (FORMAT parquet, COMPRESSION $c)\"\n e=$(date +%s.%N)\n printf '%10d bytes %6.2f s\\n' $(stat -c %s work/pq/o_$c.parquet) $(echo \"$e - $s\" | bc)\ndone\nls -l work/orders-2m.csv | awk '{printf \"%-13s%10d bytes (the CSV)\\n\", \"csv\", $5}'snappy 51477383 bytes 0.93 s\nzstd 34206903 bytes 0.59 s\ngzip 34908832 bytes 1.20 s\nuncompressed 75512763 bytes 0.68 s\ncsv 129919987 bytes (the CSV)| Codec | Size | Time |
|---|---|---|
| none | 75,512,763 bytes | 0.68 s |
| snappy (DuckDB's default) | 51,477,383 bytes | 0.93 s |
| gzip | 34,908,832 bytes | 1.20 s |
| zstd | 34,206,903 bytes | 0.59 s |
| the CSV | 129,919,987 bytes |
Even without compression the Parquet file is 58% of the CSV, because numbers stored as binary integers and doubles take less space than their digits. Compression takes it to 26% with zstd. zstd was both the smallest and the fastest to write in this run, which is not a general law; compression ratios depend on the data. The same job through other libraries:
/usr/bin/time -f \"pandas + pyarrow: %e s, peak %M KB\" python -c \"\nimport pandas as pd\npd.read_csv('work/orders-2m.csv').to_parquet('work/pq/o_pandas.parquet')\n\"\nls -l work/pq/o_pandas.parquet | awk '{print $5 \" bytes\"}'\n/usr/bin/time -f \"polars streaming: %e s, peak %M KB\" python -c \"\nimport polars as pl\npl.scan_csv('work/orders-2m.csv').sink_parquet('work/pq/o_polars.parquet')\n\"\nls -l work/pq/o_polars.parquet | awk '{print $5 \" bytes\"}'pandas + pyarrow: 2.82 s, peak 500296 KB\n49004545 bytes\npolars streaming: 1.12 s, peak 531992 KB\n35534323 bytespandas with pyarrow read the CSV into memory and wrote 49,004,545 bytes in 2.82 s with a 500,296 KB peak. polars' streaming sink did it in 1.12 s with 531,992 KB for 35,534,323 bytes. They are fine for this size and need the data in memory (pandas) or a streaming plan (polars); DuckDB's COPY is the shortest to write. For files larger than memory, see reading a large CSV in chunks and the DuckDB articles.
A Parquet file is a sequence of row groups, and each row group stores each column as its own block. DuckDB can show the file's metadata:
\nduckdb -box -c \"SELECT path_in_schema AS col, type, compression, encodings, total_compressed_size AS bytes, stats_min, stats_max FROM parquet_metadata('work/pq/o_snappy.parquet') WHERE row_group_id = 0\"\nduckdb -box -c \"SELECT count(DISTINCT row_group_id) AS row_groups, max(row_group_num_rows) AS rows_per_group FROM parquet_metadata('work/pq/o_snappy.parquet')\"┌─────────────┬────────────┬─────────────┬──────────────────┬────────┬─────────────────────┬─────────────────────┐\n│ col │ type │ compression │ encodings │ bytes │ stats_min │ stats_max │\n├─────────────┼────────────┼─────────────┼──────────────────┼────────┼─────────────────────┼─────────────────────┤\n│ order_id │ INT64 │ SNAPPY │ PLAIN │ 492545 │ 1 │ 122880 │\n│ customer_id │ INT64 │ SNAPPY │ PLAIN │ 583235 │ 1 │ 249998 │\n│ zip │ BYTE_ARRAY │ SNAPPY │ PLAIN │ 669825 │ 00500 │ 99997 │\n│ amount │ DOUBLE │ SNAPPY │ PLAIN_DICTIONARY │ 339061 │ 0.08 │ 607.8 │\n│ created_at │ INT64 │ SNAPPY │ PLAIN │ 911679 │ 2025-01-01 00:03:30 │ 2025-12-31 23:55:54 │\n│ status │ BYTE_ARRAY │ SNAPPY │ PLAIN_DICTIONARY │ 46672 │ failed │ shipped │\n│ country │ BYTE_ARRAY │ SNAPPY │ PLAIN_DICTIONARY │ 46655 │ BR │ US │\n│ note │ BYTE_ARRAY │ SNAPPY │ PLAIN_DICTIONARY │ 41200 │ call first │ rush order │\n└─────────────┴────────────┴─────────────┴──────────────────┴────────┴─────────────────────┴─────────────────────┘\n┌────────────┬────────────────┐\n│ row_groups │ rows_per_group │\n├────────────┼────────────────┤\n│ 17 │ 124226 │\n└────────────┴────────────────┘The 2,000,000 rows are in 17 row groups of up to about 122,880 rows. For the first group, each column has its own type (INT64, BYTE_ARRAY for strings, DOUBLE), a compression codec (SNAPPY), an encoding and a size in bytes. Four things to read in it.
Column layout. Because amount is stored apart from note, a query that needs amount reads only the amount blocks. A CSV cannot do that; its columns are interleaved in every line, and a reader has to scan them all to find the fourth field in each.
Types. order_id is a 64-bit integer, 8 bytes for every value (the first group's block is 492,545 bytes, about 4 bytes a value after compression). The reader does not parse digits; it reads numbers.
Encoding. status, country and note use PLAIN_DICTIONARY: the file stores each distinct value once and the column as a list of small indexes. status has four distinct values, and its whole block for 122,880 rows is 46,672 bytes, about 3 bits a row. The text of those words costs 6 to 9 bytes a row in the CSV.
Statistics. Each row group records the minimum and maximum of each column. For order_id in the first group they are 1 and 122,880; for status they are failed and shipped. A query with WHERE order_id BETWEEN 1000 AND 1100 can read the statistics and skip every row group whose range does not include the values. A CSV has no statistics; to know where order 1000 is, you read the file.
Three queries, run against the CSV (read with read_csv, which re-parses on every run) and against the Parquet file:
Q1=\"SELECT status, count(*) AS n, round(sum(amount), 2) AS total FROM %s GROUP BY 1 ORDER BY 1\"\nfor src in \"read_csv('work/orders-2m.csv')\" \"'work/pq/o_snappy.parquet'\"; do\n echo \"== $src\"\n q=$(printf \"$Q1\" \"$src\")\n time duckdb -c \"$q\" | head -n 12 | tail -n 4\ndone== read_csv('work/orders-2m.csv')\n│ pending │ 400329 │ 32077864.5 │\n│ refunded │ 400135 │ 32069815.84 │\n│ shipped │ 399890 │ 32018038.17 │\n└──────────┴────────┴─────────────┘\n\nreal\t0m0.202s\nuser\t0m0.395s\nsys\t0m0.131s\n== 'work/pq/o_snappy.parquet'\n│ pending │ 400329 │ 32077864.5 │\n│ refunded │ 400135 │ 32069815.84 │\n│ shipped │ 399890 │ 32018038.17 │\n└──────────┴────────┴─────────────┘\n\nreal\t0m0.070s\nuser\t0m0.084s\nsys\t0m0.047sfor src in \"read_csv('work/orders-2m.csv')\" \"'work/pq/o_snappy.parquet'\"; do\n echo \"== $src: one column, max(amount)\"\n time duckdb -noheader -list -c \"SELECT max(amount) FROM $src\"\ndone\necho; echo \"== a selective filter\"\nfor src in \"read_csv('work/orders-2m.csv')\" \"'work/pq/o_snappy.parquet'\"; do\n echo \"$src\"\n time duckdb -noheader -list -c \"SELECT count(*) FROM $src WHERE order_id BETWEEN 1000 AND 1100\"\ndone== read_csv('work/orders-2m.csv'): one column, max(amount)\n654.24\n\nreal\t0m0.203s\nuser\t0m0.323s\nsys\t0m0.137s\n== 'work/pq/o_snappy.parquet': one column, max(amount)\n654.24\n\nreal\t0m0.036s\nuser\t0m0.026s\nsys\t0m0.013s\n\n== a selective filter\nread_csv('work/orders-2m.csv')\n101\n\nreal\t0m0.191s\nuser\t0m0.334s\nsys\t0m0.127s\n'work/pq/o_snappy.parquet'\n101\n\nreal\t0m0.041s\nuser\t0m0.031s\nsys\t0m0.021s| Query | CSV | Parquet |
|---|---|---|
GROUP BY status with count and sum(amount) | 0.202 s | 0.070 s |
max(amount) (one column) | 0.203 s | 0.036 s |
count(*) WHERE order_id BETWEEN 1000 AND 1100 | 0.191 s | 0.041 s |
A factor of three to six, with a caveat: these queries are fast on the CSV too. DuckDB's CSV reader is parallel and the file was in the operating system's cache, so 130 MB is parsed in 0.2 s. On a wider table, a larger file, a cold cache or a slower reader the gap grows, because the CSV cost scales with all the bytes and the Parquet cost with only the columns and row groups the query touches. A longer comparison measures that on a bigger file. For a table you query once, converting costs more than it saves; for one you query repeatedly, you pay once.
\nDuckDB made amount a DOUBLE, and a double cannot hold 103.10 exactly. For money, a decimal type is the right one, and the conversion can say so:
duckdb -c \"COPY (SELECT order_id, customer_id, zip, CAST(amount AS DECIMAL(10,2)) AS amount, created_at, status, country, note FROM read_csv('work/orders-2m.csv')) TO 'work/pq/o_decimal.parquet' (FORMAT parquet, COMPRESSION zstd)\"\nls -l work/pq/o_decimal.parquet work/pq/o_zstd.parquet | awk '{printf \"%-22s %d bytes\\n\", $9, $5}' | sed 's#work/pq/##'\nduckdb -box -c \"SELECT column_name, column_type FROM (DESCRIBE SELECT * FROM 'work/pq/o_decimal.parquet') WHERE column_name = 'amount'\"\nduckdb -box -c \"SELECT amount FROM 'work/pq/o_decimal.parquet' WHERE order_id = 1\"o_decimal.parquet 33922831 bytes\no_zstd.parquet 34206903 bytes\n┌─────────────┬───────────────┐\n│ column_name │ column_type │\n├─────────────┼───────────────┤\n│ amount │ DECIMAL(10,2) │\n└─────────────┴───────────────┘\n┌────────┐\n│ amount │\n├────────┤\n│ 103.10 │\n└────────┘With CAST(amount AS DECIMAL(10,2)) the Parquet column is DECIMAL(10,2), the value for order 1 reads back as 103.10, and the zstd file is 33,922,831 bytes against 34,206,903 for the DOUBLE version. It is smaller, because a fixed-point decimal compresses better than 8-byte floats with irregular bits. Give identifiers, money and codes their types explicitly, pass the column types to the reader (types= in DuckDB, dtype= in pandas) instead of letting it guess, and keep the choice in the conversion script. The CSV types article explains why the guess is a guess.
Round-trip the data and compare it as sets of rows:
\nduckdb -noheader -list -c \"SELECT count(*) FROM (SELECT * FROM read_csv('work/orders-2m.csv') EXCEPT SELECT * FROM 'work/pq/o_snappy.parquet')\"\nduckdb -noheader -list -c \"SELECT count(*) FROM (SELECT * FROM 'work/pq/o_snappy.parquet' EXCEPT SELECT * FROM read_csv('work/orders-2m.csv'))\"0\n0EXCEPT in both directions returned 0, so the CSV and the Parquet file hold the same 2,000,000 rows under the inferred types. It is a strong check for types and values at once, and it would fail if a zip had lost its leading zero or a timestamp had shifted. Also compare the row counts and keep the CSV's checksum with the Parquet file.
Parquet is not readable in a text editor; you need a tool, and it is awkward to inspect on a machine without one. Appending a row means writing a new file, since the layout is immutable, so a stream of small writes produces a pile of small files that reads slowly; the usual answer is to batch and compact. Changing a column's type means rewriting. Not every tool reads it: older ETL software, some spreadsheets and plenty of scripts want CSV. And the format has its own complexity: row group size, codecs and dictionary limits all matter once the data is big.
\nCompare row counts and run the EXCEPT check. Look at the schema and confirm each identifier is text and each amount is a decimal. Query a few values and compare them with the CSV. Time one query you care about against both files, with a cold cache if you can, and decide whether the saving is worth the extra file. Note the conversion script and the source checksum, so the Parquet can be rebuilt.
For a file that goes to a person, to a partner who does not use analytical tools, to a system that only speaks CSV, or into a repository where a diff has to be readable, keep CSV. For a table that is read once, or is small enough that the parse time does not register, the conversion buys nothing. For a data set that will be queried many times by columns and filters, convert it, put the types in the script and keep the CSV as the source. The large-CSV hub puts this in the context of the other options.
","date_published":"2026-11-05T15:09:00+00:00","tags":["CSV"]},{"id":"https://datahacks.dev/library/xml-good","url":"https://datahacks.dev/library/xml-good","title":"What XML Is Good At and Why It Still Shows Up in Your Data","summary":"A tested beginner guide to XML: where you will meet it in data work, what it does better than JSON, how it is structured, and the first checks to run.","content_html":"You did not choose XML. It was in the spreadsheet you opened, the sitemap your site publishes, the feed a vendor sends every night, the config file of your build tool and the export that arrived with the words \"legacy system\" attached. XML is thirty years old, and it holds more of the world's structured information than any newer format. If you work with data, you will be asked to read it, and the first hour is the worst, because every tutorial assumes you already know what it is for.
\nThis guide is the orientation I would want: what XML is good at, where it hides, how it is structured, what goes wrong first, and where to go next. The examples are small files I ran through real tools, with the numbers the tools printed.
\nXML, the Extensible Markup Language, is a way to write a tree as text. Everything in a document is one of a small set of things.
\nAn element is a named node, written with a start tag and an end tag: <price>12.50</price>. Elements nest, and the nesting is the tree. A document has exactly one outermost element, the root.
An attribute is a name and a value attached to an element's start tag: <price currency=\"USD\">. It is one string, single per name, with no structure of its own.
Text is what is between the tags. Comments, processing instructions and CDATA sections are the rest. And the whole is usually opened by a declaration that names the version and the character encoding.
\n<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<catalog>\n <book id=\"b1\" lang=\"en\">\n <title>Cache Patterns</title>\n <price currency=\"USD\">12.50</price>\n </book>\n</catalog>That is all there is. The rest is conventions and standards built on top of this one idea: a document is a tree of named, ordered nodes, and any tool that can read the tree can read any XML. The grammar is strict, and a parser reads or refuses a document, which is why the same format has worked for decades.
\nCompared with JSON, which is the format most people reach for now, XML has strengths that are real and specific.
\nIt handles text with markup. In <p>Hello <b>big</b> world</p> the bold word sits in the middle of a sentence. XML represents that exactly. JSON has no good way to. This is why books, articles, legal texts, manuals and web pages are marked up documents, not records, and why the formats that carry them (DocBook, DITA, EPUB, XHTML) are XML.
Order is data. Children appear in a defined sequence and a parser preserves it. A list of steps, a table of rows or a sentence of words has an order that matters, and XML keeps it without any convention.
\nIt has namespaces. Two vocabularies can share one document without their element names colliding. An Atom feed can carry Dublin Core metadata and a media extension at once, each element tagged with the vocabulary it comes from (how namespaces work).
\nIt has mature validation. A DTD or an XML Schema can state what a document must look like, down to types and enumerations, and a tool checks it (well formed versus valid). Schematron adds business rules. Industries built contracts on that.
\nIt has a toolchain for transformation and query. XPath selects nodes, XSLT turns one XML document into another document or into text, and XQuery queries collections. They are standard and have been stable for twenty years.
\nIt carries comments and processing instructions. A config file can explain itself, and a document can carry directions to the program that reads it.
\nIts weaknesses are the mirror image. It has no native numbers, booleans, null or arrays, so every value is text until a schema says otherwise. A list of one item is indistinguishable from a single item without the schema. It is verbose: the same 50,000 records were 6.40 MB as XML and 4.38 MB as JSON, and parsed about twice as slowly in Python (XML versus JSON for data exchange has the numbers and what a conversion loses). And it has traps for the careless: namespaces, entities and encodings.
\nThe quickest way to see why XML persists is to open some files. Each of these is a format you probably use already.
\nAn .xlsx file is a zip archive. I generated a workbook with 2,000 rows and listed it:
[Content_Types].xml 1167 bytes\n_rels/.rels 587 bytes\nxl/_rels/workbook.xml.rels 697 bytes\nxl/worksheets/sheet1.xml 243612 bytes (compressed 30946)\nxl/workbook.xml 550 bytes\nxl/sharedStrings.xml 1611 bytes\nxl/styles.xml 867 bytes\nxl/theme/theme1.xml 6994 bytes\ndocProps/core.xml 592 bytes\ndocProps/app.xml 784 bytesTen parts, every one of them XML. The data is in sheet1.xml, which has a row element per row and a c (cell) element per cell:
<row r=\"2\" spans=\"1:4\"><c r=\"A2\"><v>1</v></c><c r=\"B2\" t=\"s\"><v>4</v></c><c r=\"C2\"><v>1.5</v></c></row>The attribute t=\"s\" means the value is an index into sharedStrings.xml, which holds each distinct string once (55 unique strings for 2,289 uses). The root element is in the namespace http://schemas.openxmlformats.org/spreadsheetml/2006/main, which is the thing to know before you write an XPath against it. The 2,000 rows made 243,612 bytes of XML that compressed to 30,946 inside the zip.
That structure is the reason you can read a workbook without Excel. At 100,000 rows the sheet was 13.91 MB of XML in a 1.7 MB file. Reading it by streaming the XML with lxml.iterparse and summing a column took 0.64 s, against 1.72 s for openpyxl in read-only mode and 2.12 s for pandas.read_excel, which sits on top of an XML reader. The XML is not a thing to be afraid of. It is the fastest way into the file if you know what to look for, and the same structure holds for Word (.docx) and PowerPoint (.pptx).
A sitemap, the file that tells search engines your URLs, is XML in a namespace:
\n<urlset xmlns=\"http://www.sitemaps.org/schemas/sitemap/0.9\">\n <url><loc>https://example.com/</loc><lastmod>2026-01-02</lastmod></url>\n</urlset>An RSS or Atom feed is the same kind of file, with item or entry elements. An SVG image is XML in the SVG namespace, where a circle is <circle cx=\"32\" cy=\"32\" r=\"30\"/> and a shape's style is an attribute. All three were well formed under xmllint, and I read a value out of each with an XPath: the two sitemap URLs, the title First & best of the feed item (note that the file says &, which the parser turns back into an ampersand), and 64 for the icon's width.
SOAP web services, banking and insurance messages, healthcare documents and customs declarations are XML, defined by schemas that standards bodies publish. A request looks like this:
\n<soap:Envelope xmlns:soap=\"http://schemas.xmlsoap.org/soap/envelope/\" xmlns:m=\"urn:example:orders\">\n <soap:Body><m:GetOrder><m:Id>7</m:Id></m:GetOrder></soap:Body>\n</soap:Envelope>Two namespaces, one for the envelope and one for the application. You will meet this when you integrate with a system that is older than your company. The contract is a schema file, and the first thing to do is validate your messages against it.
\nMaven's pom.xml, Android layouts and manifests, .NET project files, Spring configuration, Ant builds and macOS property lists are XML. The pom.xml I used declares a default namespace and an xsi:schemaLocation attribute that points to its schema, and its dependencies were readable with an XPath on the local names (artifactId returned app and jackson-databind). Config in XML has comments and a schema, which is why the format lasted there.
Scientific data (GPX and KML for tracks and maps, CML for chemistry, SBML for biology), publishing (EPUB, DocBook, JATS for journals), finance (XBRL for company reports, FpML), government and legal data (AKN, XBRL again) and countless exports from software written between 1998 and 2015. If a document has structure that needs to be validated and kept for decades, someone probably chose XML for it.
\nWhen an XML file arrives, three checks save most of the pain.
\nIs it well formed? xmllint --noout file.xml prints nothing if it is, and one error line if it is not. The rules are short (one root, matched tags, quoted attributes, escaped & and <), and the error names the line. The cheat sheet of syntax rules lists 51 cases with their messages, and fixing XML parse errors fast goes through the failures you will see.
Does the encoding match the bytes? The declaration says what the file claims. A file whose bytes are UTF-8 but whose declaration says ISO-8859-1 parses without an error and gives café (the wrong encoding problem). Compare the declaration with what file reports.
What namespace is it in? Look at the root element for xmlns. If there is a default namespace, every unprefixed element is in it, and a path like //entry finds nothing until you map a prefix. This single fact explains most \"my XPath returns nothing\" questions (namespaces).
After that, look at the structure. The XML formatter shows a document indented, and xmllint --format file.xml does the same on the command line. Find the repeating element (row, item, order): that is your record.
There are three levels of tool, and you will use all of them.
\nQuery it where it is. XPath selects nodes: //item[price>10]/title. In the command line xmllint --xpath and in code lxml's .xpath() do it. Ten expressions cover most needs (XPath queries that cover most real needs), and the XPath tester lets you try one on a sample.
Read it in code. Python's standard library has xml.etree.ElementTree, and lxml is the faster and fuller library (parsing and writing XML in Python). For large files, stream with iterparse and clear nodes as you go, which handled a 14 MB sheet in well under a second (parsing large XML files with iterparse). For untrusted input, use defusedxml (safe XML parsing in Python).
Load it into a database. PostgreSQL has an xml type and xmltable, SQL Server has nodes() and value(), Oracle has XMLTABLE, and MySQL has ExtractValue and LOAD XML (storing and querying XML inside a database). When the destination is tables, the pattern is to find the repeating element and turn each into a row (turning XML feeds into relational tables).
A short list of what bites newcomers, each covered in a spoke.
\n**A raw & or < in text** from string concatenation. Use a serializer (escaping special characters).
A path that finds nothing because of a default namespace.
HTML entities like in XML, which are undefined.
Treating indentation as data. Whitespace between elements is text in the tree (why whitespace sometimes matters).
Trusting the encoding declaration.
Parsing untrusted XML with a parser that resolves entities. A DTD can read files or expand a few hundred bytes into gigabytes (what a DTD is). Reject documents that carry a DOCTYPE unless you need one.
Assuming a list. An element that repeats is a list; one that appears once is not, unless you handle it.
If you are designing a new format for records between systems you control, use JSON. It is smaller, faster and simpler. If your content is documents with inline structure, a long-lived archive with strict validation, a vocabulary shared by many parties, or an existing standard, use XML. When a counterparty decides for you, learn to read it well; it is easier than its reputation.
\nThe articles on this site that follow the guide fall into five groups. The basics (this guide plus the nine that follow it) explain the syntax, validity, attributes, namespaces, escaping, whitespace, encodings, DTDs and oddities. Debugging (fixing XML parse errors fast) works through each error message and its cause. XPath (XPath queries that cover most real needs) covers selecting nodes. Python (parsing and writing XML in Python) shows the libraries. And the database and pipeline groups show how to get XML into tables.
\nThe primary references are the XML 1.0 recommendation, which is short and readable, and the Namespaces in XML recommendation. The libxml2 tools used above are documented at the GNOME libxml2 wiki.
\nXML came out of SGML, the heavyweight markup language used for technical publishing, and the W3C published XML 1.0 as a recommendation in 1998. The goal was a simpler SGML that could travel on the web: strict enough for programs to parse with no guessing, flexible enough to define any vocabulary. Within a few years it was everywhere, as the format for configuration, for web services (SOAP and its relatives), for office documents and for data exchange between companies.
\nThen JSON arrived and took the part of the job that was always awkward in XML: sending records between a browser and a server. For that, JSON is smaller, simpler and closer to the data structures programs already use. XML did not go away, because it was never only a record format. The standards written in it were expensive to write and are expensive to replace: tax filings, healthcare messages, publishing pipelines, mapping data, accounting formats. The tools (parsers, validators, XSLT engines) are stable, free and in every language. A format that works and is already specified in your industry is hard to dislodge.
\nSo the practical position is: you will not often choose XML for new work, and you will often be handed it.
\nThe workbook example is worth doing once with your own hands, because it shows the whole workflow in a few lines. A sheet is a stream of row elements. Streaming them with iterparse, and clearing each row once it has been handled, keeps memory flat however many rows there are:
M = \"{http://schemas.openxmlformats.org/spreadsheetml/2006/main}\"\nwith zipfile.ZipFile(\"big.xlsx\").open(\"xl/worksheets/sheet1.xml\") as f:\n for event, el in etree.iterparse(f, tag=M + \"row\"):\n for c in el:\n if c.get(\"r\", \"\").startswith(\"C\"):\n total += float(c.find(M + \"v\").text)\n el.clear()\n while el.getprevious() is not None:\n del el.getparent()[0]Three things in it are the lessons of this guide in miniature. The tag is written with the namespace in curly braces, because the elements are in the spreadsheet namespace. The loop reads cells by their r attribute (C2, C3) because the document is a tree of named things and not a grid. And the clearing at the end keeps the tree from growing: without it the same loop would hold the whole sheet in memory. For my 100,000 row file the sum of the third column came out as 5,000,000.0 in 0.64 s.
Note that strings in a workbook may be stored inline or in sharedStrings.xml, depending on the program that wrote the file, and a cell with t=\"s\" holds an index, not the text. If you are reading real files from Excel and not generated ones, reach for a library (openpyxl, pandas) and keep the raw approach for when the library is too slow or too memory-hungry.
The usual first task is turning an XML feed into a table. With the sitemap and the RSS feed from above, it took a few lines:
\nns = {\"s\": \"http://www.sitemaps.org/schemas/sitemap/0.9\"}\nrows = [{\"loc\": u.findtext(\"s:loc\", namespaces=ns),\n \"lastmod\": u.findtext(\"s:lastmod\", namespaces=ns),\n \"changefreq\": u.findtext(\"s:changefreq\", namespaces=ns)}\n for u in tree.iterfind(\"s:url\", ns)]\npd.DataFrame(rows) loc lastmod changefreq\n https://example.com/ 2026-01-02 NaN\nhttps://example.com/about NaN monthlyTwo points show up immediately. Optional elements become missing values (lastmod is absent on the second URL, so findtext returned None), which is the XML way of saying \"not provided\". And everything is text: the RSS publication date Fri, 02 Jan 2026 03:04:05 GMT needed an explicit format string to become a timestamp, 2026-01-02 03:04:05+00:00. Pandas can do the whole thing in one call, pd.read_xml(\"sitemap.xml\", xpath=\".//s:url\", namespaces=ns), which gave the same table with the priority column included. Without the namespace mapping it raised ValueError: xpath does not return any nodes: the default-namespace trap in its most common form.
That is the whole shape of the work. Find the repeating element, name its namespace, take the fields you want, convert the types, and decide what a missing element means.
\nNumbers from the other articles in this cluster give a feel for the practical costs. The same 50,000 orders took 3.83 MB written with attributes, 5.88 MB with child elements and 7.43 MB indented, and all of them compressed to between 0.25 and 0.28 MB, so size matters in memory and on disk but not on the wire (attributes versus child elements). Parsing them took 0.09 to 0.32 s depending on the parser and the layout. The parse of a 6.4 MB document took 0.109 s in lxml and 0.174 s in ElementTree. Neither is slow, and both are slower than the equivalent JSON.
The costs that hurt are not speed. They are the hours spent on a namespace, an encoding or an entity. The cheat sheet, the encoding article and the entity article exist for those.
\nXML parsers have a history of vulnerabilities, and the cause is a feature: the DTD. A document can declare entities, and an entity can point at a file (file:///etc/hostname) or a URL, or expand a few hundred bytes into hundreds of thousands of characters. Current ElementTree and lxml refuse to read external files by default and cap entity expansion: an eight-level expansion was rejected by both. A parser with entity resolution switched on read the file's contents, and defusedxml rejected every document that declared an entity at all.
The rule for anything from outside your own systems is to use a parser configured for untrusted input, to reject documents with a DOCTYPE, and to limit size and depth before parsing, as for any format (what a DTD is and why you rarely need one, safe XML parsing in Python). The same attitude toward untrusted input that applies to JSON (JSON security mistakes in apps and pipelines) applies here, and XML adds the entity features to the list.
A handful of terms recur in every XML conversation, and knowing them makes documentation readable.
\nA node is any item in the tree: an element, an attribute, a piece of text, a comment. The root is the single outermost element. Well formed means it obeys the syntax. Valid means it also follows a schema.
\nA namespace is a URI that scopes names, and a prefix is a local alias for it. The URI is the identity and the prefix is a convenience. A default namespace is one declared without a prefix, which applies to unprefixed elements.
\nA DTD is the old schema language, and XSD (XML Schema) the newer one, written in XML itself. Schematron states rules as XPath assertions. RELAX NG is a third, less common alternative.
\nXPath is the language for selecting nodes. XSLT transforms a document by applying templates, and XQuery queries collections the way SQL queries tables.
\nDOM is the model where a parser loads the whole tree into memory and you walk it. SAX and StAX are streaming models that report events as the parser reads, so memory stays small. ElementTree and lxml are DOM-like with a streaming option (iterparse) that gives you the middle path: events for each element, with the subtree available while you work on it.
An entity is a named piece of text, and a character reference (©) is a numeric way to write a character. CDATA is a section where markup characters are literal.
Finally, canonical XML (c14n) is a defined way to write a document so that two logically equal documents have identical bytes. It is what digital signatures use, and what you would use to compare two documents byte for byte if layout differences should not count.
\nNone of these needs mastering before you start. They are the words that explain an error message, and now you can look each one up.
\nIf I had one week to become useful with XML, I would spend the first day on XPath, because selecting the right nodes is most of the job. The second day goes to a library in the language I use, reading, writing and streaming. The third day is for namespaces and encodings, which are where files go wrong. The fourth is validation: running a file against a supplied XSD with xmllint and reading the errors. The fifth day is for getting data into the place it is needed, tables or JSON, with a plan for missing elements and repeated ones. After that, XSLT is worth a look if you convert documents often, and Schematron if you own a format. Everything on that list has an article here, in this cluster, with the commands run and the output shown.
If you are new, read in this order: the syntax cheat sheet, well-formed versus valid, namespaces, escaping, and the encoding article. Those five remove most of the first-day problems. Then take whichever of debugging, XPath or Python matches what you are doing, and come back to the database articles when you need to load the data into tables. Each article is one idea with one or two experiments you can repeat; every command above ran as shown.
\nYou are designing a record exchange between your own services. JSON is smaller, faster to parse and has the types you want. Do not take on namespaces and schemas without a reason.
\nThe data is tabular. A CSV or a Parquet file is the right shape for rows and columns. XML would add size and a mapping step. If you receive XML that is really a table, convert it once and work with the table.
\nYou need human-edited configuration. YAML and TOML are friendlier to type than XML for small configs, though XML offers comments and validation too.
\nSpeed and size are the constraint. Binary formats and JSON Lines beat XML for high-volume data. Keep XML for the cases where its features are the point.
","date_published":"2026-11-05T00:17:00+00:00","tags":["XML"]},{"id":"https://datahacks.dev/library/inside-excel-file","url":"https://datahacks.dev/library/inside-excel-file","title":"What Is Inside an Excel File and What It Means for Data","summary":"See what an XLSX workbook really contains, from cell types and shared strings to dates, formulas, styles and limits, and what each part means for your data.","content_html":"An Excel file looks like a grid, and most people treat it as one: rows, columns, values. A program that reads the file sees something different. It sees a package of XML documents in which the grid is only one part, where a cell may hold a pointer to a string stored elsewhere, where a date is a number in disguise and a formula is a recipe plus a remembered answer. Nearly every Excel data problem you will meet comes from the gap between the grid and the package: the zip code that lost its leading zero, the date that arrives as 45292, the total that is empty when a script reads it, the column that is text in one row and a number in the next.
\nThis guide opens the package and explains what is in it, in the order that matters for people who move data in and out of spreadsheets. It is the map for the Excel cluster on this site. Each section points to a focused article, and the later clusters on loading Excel data into a database, sending database results to Excel, openpyxl, pandas, dates and types and translating formulas into SQL build on it.
\nRename report.xlsx to report.zip and open it with any archive tool. It opens, because an XLSX file is a ZIP archive. Inside are folders and XML files arranged according to the Open Packaging Conventions, a standard for bundling the parts of a document that Microsoft Office, and several other formats, share. A small workbook contains roughly this:
[Content_Types].xml\n_rels/.rels\ndocProps/core.xml\ndocProps/app.xml\nxl/workbook.xml\nxl/_rels/workbook.xml.rels\nxl/worksheets/sheet1.xml\nxl/sharedStrings.xml\nxl/styles.xml\nxl/theme/theme1.xmlEach file has a job. [Content_Types].xml declares what type each part is. The _rels folders hold relationships, which tell a reader that sheet 1 is the file at xl/worksheets/sheet1.xml. xl/workbook.xml lists the sheets with their names, visibility and the date system. xl/worksheets/ holds one XML file per sheet with the cell data. xl/sharedStrings.xml holds the text values. xl/styles.xml holds fonts, fills, borders and, importantly for data, the number formats. Larger workbooks add folders for tables, charts, drawings, images, comments, defined names and a calculation chain.
The practical consequence is that nothing about the file is hidden. If an import tool misreads a value, you can unzip the file and look at the XML the tool read. The article an XLSX file is a zip of XML walks through doing exactly that, and a short script gives a quick inventory:
\nimport zipfile\n\nwith zipfile.ZipFile(\"report.xlsx\") as z:\n for info in z.infolist():\n print(f\"{info.file_size:>10} {info.compress_size:>10} {info.filename}\")The two size columns show the uncompressed and compressed size of each part. For a workbook with a lot of data, the sheet files dominate. For one that is oddly large, a media folder or a huge styles.xml is the usual explanation, as covered in why Excel files get huge.
The older .xls format is not a zip and not XML. It is a binary format from before 2007, and it has tighter limits. The differences between the three formats you will meet, .xls, .xlsx and .xlsb, are in XLS versus XLSX versus XLSB.
Open xl/worksheets/sheet1.xml and you find a sheetData element that holds rows, each holding cells. Here is a minimal example with a header and two data rows:
<worksheet>\n <sheetData>\n <row r=\"1\">\n <c r=\"A1\" t=\"s\"><v>0</v></c>\n <c r=\"B1\" t=\"s\"><v>1</v></c>\n <c r=\"C1\" t=\"s\"><v>2</v></c>\n </row>\n <row r=\"2\">\n <c r=\"A2\" t=\"s\"><v>3</v></c>\n <c r=\"B2\"><v>19.99</v></c>\n <c r=\"C2\" s=\"1\"><v>45292</v></c>\n </row>\n <row r=\"3\">\n <c r=\"A3\" t=\"s\"><v>4</v></c>\n <c r=\"B3\"><v>5</v></c>\n <c r=\"C3\" s=\"1\"><v>45293</v></c>\n </row>\n </sheetData>\n</worksheet>A cell is a c element with a reference r such as B2, an optional type t, an optional style index s, and a value v. Three things deserve attention.
First, only cells that hold something are written. An empty cell has no element at all, so the file does not store a rectangle. A reader must decide what to do about the gaps, and different libraries return None, NaN or an empty string. Row elements are skipped the same way when the whole row is empty.
Second, the type attribute is small. With no t, the value is a number. t=\"s\" means the value is an index into the shared strings table. t=\"b\" is a boolean, t=\"e\" an error such as #DIV/0!, t=\"str\" a string that came from a formula, and t=\"inlineStr\" a string stored right in the cell, which some writers prefer. There is a t=\"d\" for ISO dates that Excel itself rarely writes. There is no text type separate from strings, and there is no date type.
Third, the style index s points into styles.xml. The date cell C2 holds the plain number 45292 and a style with index 1, and that style says \"format as a date\". Strip the style and the cell is a number.
Everything a program reads from a sheet comes from these elements. The article cheat sheet of Excel data types as programs see them lists each case with what openpyxl and pandas return for it.
\nThe sample above shows <v>0</v> for the header cells, with t=\"s\". The zero is not a value; it is a position in the shared strings table:
<sst count=\"5\" uniqueCount=\"5\">\n <si><t>product</t></si>\n <si><t>price</t></si>\n <si><t>ordered</t></si>\n <si><t>notebook</t></si>\n <si><t>pen</t></si>\n</sst>Each unique string is stored once, and each cell that uses it points to its index. A column of one hundred thousand rows with ten distinct values stores ten strings and a hundred thousand small numbers. That saves space, and it is why a reader must load the whole shared strings table before it can interpret any cell. It is also why streaming readers need care: they hold the strings in memory even when they stream the cells (reading large Excel files in read-only mode).
\nA string that has partial formatting, such as one bold word, is stored as rich text with several runs. Most data libraries flatten it to plain text and discard the formatting. A string can hold up to 32,767 characters, and a cell that holds more is truncated by Excel on save.
\nOne more consequence: text that looks like a number is still text if the cell says so. The zip code 01234 typed into a cell formatted as text is stored as a shared string, and it stays 01234. Typed into a general cell, Excel converts it to the number 1234 on entry, and the zero is gone before any file is written (keeping leading zeros).
A number cell stores an IEEE 754 double-precision value. This gives about 15 to 17 significant decimal digits, and Excel displays and keeps only 15. Anything beyond is rounded when the number is typed. A 16-digit credit card number typed as a number loses its last digit to zero, and nothing recovers it. Large identifiers belong in text cells (Excel number precision and the 15 digit limit). Integers are exact up to 2 to the power 53, which is 9,007,199,254,740,992, but the 15-digit display and entry limit applies first.
\nIntegers and decimals are the same type. The value 5 and the value 5.0 are one number, stored as 5. A reader library decides whether to hand your code an integer or a float, and the choice differs. openpyxl returns an int when the stored text has no decimal point and a float otherwise; pandas often returns floats for a whole column if any cell has a decimal or is empty. A column of ids with one missing value arrives as floats such as 17.0 unless you ask for a nullable integer type.
Percentages, currency and scientific notation are all formats on top of the same number. A cell shows 45% while the stored value is 0.45. A cell shows $1,234.50 while the stored value is 1234.5. The format lives in styles.xml, and a program that reads the value gets 0.45 and 1234.5, which is usually what you want. It is a surprise only when you compare a screenshot with the output.
In the sheet XML above, C2 holds 45292 with a style. Excel stores a date as the number of days since an epoch, with the time of day as the fraction of a day. The value 45292 is 1 January 2024 in the default Windows date system. The value 45292.5 is noon on that day. The number format on the cell, a custom format like yyyy-mm-dd, makes it display as a date. If you change the format to General, the cell shows 45292.
This matters whenever data leaves Excel. A reader that does not look at the number format sees only a number, and your table gets a column of five-digit integers. Libraries differ: openpyxl checks the format and returns a Python datetime for cells whose format looks like a date, while a raw read of the XML gives you the number. When a date column arrives as numbers, the fix is a conversion, covered in how Excel stores dates as serial numbers, with versions for SQL and Python.
Two further details shape every conversion. The workbook has a date system flag in workbook.xml, date1904, which shifts every date by 1,462 days when set. Files created in very old versions of Excel for Mac use it (Mac and Windows date systems). And Excel treats 1900 as a leap year, which it was not, so serial numbers before 1 March 1900 are off by a day compared with the real calendar (the 1900 leap year bug). For dates after February 1900, a conversion that counts from 30 December 1899 is correct.
A formula cell stores both:
\n<c r=\"D2\"><f>B2*C2</f><v>99.95</v></c>The f element is the formula text, without the leading equals sign. The v element is the last value Excel calculated, the cached value. A program that reads the file normally gets one or the other depending on how it is configured. pandas reads the cached values. openpyxl returns the formula text unless you open the workbook with data_only=True, in which case it returns the cached value (reading only the values and not the formulas).
The cached value is only as fresh as the last time the file was saved by a program that calculates. A workbook written by a script that adds formulas has no cached values at all, since the script does not evaluate them. Open the file in Excel, save, and the values appear. Until then, anything that reads cached values sees empty cells in every formula column. The explanation, with the consequences for pipelines, is in what happens to formulas when code reads an Excel file, and the handling in pandas in formulas and cached values in pandas.
\nFormulas can also refer to other workbooks, volatile functions that change on every open (NOW, RAND) and ranges that grow. Treat a formula column as a derived column whose value depends on when it was last calculated, and when exactness matters, recompute the numbers yourself in SQL or Python from the inputs.
xl/styles.xml holds the number formats, fonts, fills, borders and cell style definitions. The sheet cells carry only an index. For data work the number formats are the part that matters: they are the reason a number looks like a date, a percentage, a currency or a phone number. A style can also hide a value completely: the format ;;; displays nothing at all while the value remains in the cell. A white font on a white fill does the same.
A direct consequence: what you see can differ from what you read, on purpose. A cell showing 1,235 may hold 1234.567. A cell showing 12/01/2024 may be 1 December or 12 January depending on the locale, and the stored number is the same either way. When you reconcile numbers between a spreadsheet and a database, compare the stored values and not the displayed ones.
Conditional formatting, data validation drop-downs and cell comments are stored in the sheet XML or in separate parts. Libraries ignore most of them by default. If a person encodes meaning in color, such as a red row for a canceled order, that meaning is lost in an import, because the color is a style and the style is not data. Ask for a column.
\nA sheet may contain structure that has no equivalent in a table. Merged cells are stored as a list of ranges:
\n<mergeCells count=\"1\"><mergeCell ref=\"A1:C1\"/></mergeCells>Only the top-left cell of a merged range holds a value. The other cells are empty in the XML. A report with a merged title across the top, or a merged region label that spans several rows, imports with blanks where the person saw one big cell. Fill the merged value downward or across before analysis (merged cells, hidden rows and other things that break imports, and handling merged cells with openpyxl).
\nHidden rows and columns still exist in the data, so an import brings them in. A person who hid 40 rows to tidy the view may not realize that the totals in your table include them. Filters work the same way: the filter hides rows and leaves them in the file. Sheets themselves can be visible, hidden or \"very hidden\", the last of which the Excel interface does not list. A pipeline that reads \"all sheets\" reads those too.
\nTitle rows, subtotals, blank spacer rows and notes at the bottom are not structure the format enforces; they are habits. A sheet is free-form, and a table with a header on row 1 is a convention. Treat every sheet as unstructured until you have looked at it. The import articles show how to cope: handling title rows above the header and skipping header junk with pandas.
\nExcel has two features that bring real structure. A table, created with \"Format as Table\", is stored in its own part as a named range with a header row, column names and a defined extent. Reading a table by name avoids guessing where the data ends (reading Excel tables by name with openpyxl). A defined name labels a cell or a range, such as TaxRate or Orders, and formulas can use it. Both are metadata in workbook.xml and the table parts, and neither is visible to a reader that only walks the cells. The explanation for developers is in Excel named ranges and tables.
If you control the file, ask for a table. It makes the import code shorter and removes the guess about where the data stops.
\nAn XLSX sheet holds at most 1,048,576 rows and 16,384 columns, which is the range from A1 to XFD1048576. That is about 17.2 billion cells in theory, far more than memory allows in practice. The older XLS format allowed 65,536 rows and 256 columns. A cell holds up to 32,767 characters, a sheet name up to 31 characters, and a number keeps 15 digits. Row and column limits matter when you export a large query: rows past the limit are lost or the write fails (Excel row and column limits and how to work around them). If your table is bigger than a million rows, Excel is the wrong container; use CSV or a database, or a columnar format such as Parquet.
The limits that surprise people are the practical ones. A workbook with a few hundred thousand rows and many formulas opens slowly and may run out of memory on 32-bit builds. A file with a million rows takes tens of seconds to parse in most libraries. Those figures depend on the hardware and library, and the right response is to measure with your own file.
\nCSV is plain text with no types, no styles and no formulas. Excel is a rich container with all of those. For moving data between systems, CSV's poverty is an advantage: what you write is what the other side reads, and no cell can hide a formula or a merged region. Excel wins when the audience is a person who will filter, format and annotate. The decision, with the failure modes of each side, is in Excel versus CSV for sharing data. A short version: send CSV to machines and XLSX to people, and when a machine must read an XLSX, ask for a table on one sheet.
\nEvery Python library on this site reads the same package, with different defaults:
\nopenpyxl is the reference reader and writer for XLSX. It gives you cell-level access to values, formulas, styles, merged ranges and tables (reading and writing Excel files with openpyxl).
pandas wraps a reader engine and gives you DataFrames in one call, which is the shortest route for tabular sheets (working with Excel in pandas).
DuckDB reads XLSX through its Excel extension, which is useful for querying a sheet with SQL without a Python step (loading Excel files with DuckDB).
Power Query inside Excel reads other files into Excel with a repeatable recipe (Power Query for repeatable Excel cleanups).
For loading into a database, the order of operations is nearly always the same: read the sheet as text where you can, stage it in a table with permissive types, check counts, then convert into the final table (staging Excel data in a table, validating row counts after an Excel import).
\nA library that reads a sheet applies the same small decision list to every cell, and knowing the list explains most surprises. If the cell has t=\"s\", look up the string. If it has t=\"b\", return a boolean. If it has t=\"e\", return an error marker, which pandas converts to NaN and openpyxl returns as the error text such as #N/A. If it has no type, it is a number, and now the library checks the style: when the number format is a date or time format, it converts the serial number into a date or datetime, and otherwise it keeps a number. A formula cell goes through the same steps using its cached value, unless the library was asked for formulas.
Two outcomes follow. The first is that the date decision depends on the format string, and custom formats that do not look like dates to the library leave the serial number as it is. A column that shows Jan-24 is read as a date; a column formatted with a vendor-specific pattern may not be. The second is that a column's final type is chosen after the cells are read. pandas inspects the values of a whole column and picks one type: integers with a missing value become floats, a column mixing numbers and text becomes object, and a column whose cells are all dates becomes a datetime column. Mixed columns are the frequent cause of \"it worked for last month's file\" failures, and detecting mixed types in an Excel column before import shows how to find them first.
The remedy is the same in every case: decide the type yourself. Read identifiers as text, name the date columns you expect, and convert at a point where a failure is loud. Specifying column types in read_excel shows the syntax, and the staging approach in the database articles moves the same decision into SQL.
\nWhen someone sends a workbook and asks you to load it, spend five minutes before writing code:
\nList the parts with the script above. Look for media, vbaProject.bin (macros), externalLinks and many sheets.
Open xl/workbook.xml and check the sheet names, any state=\"hidden\" attributes, and date1904.
Open the first sheet and look at the header row and the first few data rows, as the program will see them.
Search the sheet XML for <f>: count the formulas and decide whether you need values or recipes.
Search for mergeCell to see whether merged regions exist.
For any date-looking column, look at the cell's style and confirm it is a date format, not text.
Compare the row count of the XML with the count the sender expects.
Doing this once teaches you more about a file than ten trial imports. After a while you do it from memory, and the odd file stops being a surprise.
\n| Part | Where it lives | What it means for data |
|---|---|---|
| Cell values | xl/worksheets/sheetN.xml | Numbers, shared string indexes, booleans, errors, formulas |
| Text | xl/sharedStrings.xml | Each distinct string stored once |
| Number formats | xl/styles.xml | Whether a number displays as a date, percentage or currency |
| Sheet list | xl/workbook.xml | Names, visibility, date system |
| Tables | xl/tables/tableN.xml | Named range with headers and extent |
| Merged ranges | mergeCells in the sheet | Only the top-left cell has a value |
You exchange simple, well-formed data. If both sides agree on one sheet, a header row, text for identifiers and ISO dates, a library's defaults work and the internals never matter. The guide pays off when the defaults fail.
\nYou can ask for CSV. If the source system can export CSV, take it. You trade styles and formulas for a format you can read with head, and the trade is usually in your favor (saving an Excel sheet as CSV).
You use a managed import. Some tools, such as a database's import wizard or a BI product's connector, read the package for you. When they misread a column, you will be back here, but while they work, there is nothing to learn.
\nThe file is a report, not data. A formatted summary with subtotals, charts and notes is meant to be read by a person. Request the source data instead of parsing the report.
\nThe format is specified in ECMA-376, Office Open XML, and Microsoft documents the product's constraints in Excel specifications and limits. To move a small sheet's data into JSON for inspection, try the CSV to JSON converter.
","date_published":"2026-11-04T15:13:00+00:00","tags":["Excel"]},{"id":"https://datahacks.dev/library/merging-two-yaml-files-python","url":"https://datahacks.dev/library/merging-two-yaml-files-python","title":"Merging Two YAML Files in Python With a Deep Merge","summary":"dict update is shallow and replaces whole blocks. A 10-line deep merge, the list policy choice and what null means, tested on a base and an override file.","content_html":"You have a base configuration and an override: defaults plus an environment's changes, or a shared file plus a local one. Combining them looks like {**base, **override}. It is wrong in the way that matters most: it replaces whole blocks. This post shows the shallow merge fail, writes the recursive merge in ten lines, and covers the decisions a merge function must make about lists and nulls, because the answers are policy, not code.
# base.yaml\nserver: {host: db1, port: 5432, opts: {ssl: true, pool: 5}}\nfeatures: [search, export]\ndebug: false# override.yaml\nserver: {host: db2, opts: {pool: 20}}\nfeatures: [billing]\ndebug: ~The override changes the host and the pool size and leaves the port and ssl alone (by not mentioning them). It also replaces the features list and sets debug to null. The intended result keeps the port and TLS setting.
{**base, **override}shallow update: {'server': {'host': 'db2', 'opts': {'pool': 20}}, 'features': ['billing'], 'debug': None}The server key from the override replaced the base's server completely. The port 5432 and ssl: true are gone, though the override never mentioned them. dict.update and the ** merge operate on the top level only. For flat configuration it is fine. For nested configuration it silently drops settings, which is how a service ends up running on a default port in production.
A recursive function merges mappings key by key and replaces everything else:
\nimport copy\n\ndef merge(x, y, lists=\"replace\"):\n out = copy.deepcopy(x)\n for k, v in y.items():\n if isinstance(v, dict) and isinstance(out.get(k), dict):\n out[k] = merge(out[k], v, lists)\n elif isinstance(v, list) and isinstance(out.get(k), list) and lists == \"append\":\n out[k] = out[k] + v\n else:\n out[k] = copy.deepcopy(v)\n return outreplace lists: {'server': {'host': 'db2', 'port': 5432, 'opts': {'ssl': True, 'pool': 20}}, 'features': ['billing'], 'debug': None}\nappend lists : {'server': {'host': 'db2', 'port': 5432, 'opts': {'ssl': True, 'pool': 20}}, 'features': ['search', 'export', 'billing'], 'debug': None}Now server has the override's host (db2), the base's port (5432), and an opts mapping that combines both: ssl from the base, pool 20 from the override. The function recurses where both sides are mappings and takes the override's value otherwise. The deepcopy calls matter: without them the result shares nested dictionaries with the inputs, and a later change to the merged config would change the base too. This is the same sharing behavior that makes YAML aliases surprising (YAML anchors, aliases and merge keys).
Lists: replace or append. The two outputs above differ on features. Replace says the override's list is the whole list. Append says it adds to the base's. Both are legitimate and documented in different tools: configuration layering usually replaces, because \"I want exactly these features\" is the common intent, while lists of things to run or paths to search sometimes append. A third policy, merging lists of mappings by a key (name), is what Kubernetes strategic merge does and needs a key per list. Choose one, write it in the documentation, and put it in the function name or an argument. Do not mix policies silently.
Null in the override. debug: ~ overwrote the base's false with None. That is correct if null means explicitly unset. It is a surprise if the author wrote a key with no value as a placeholder and expected \"no change\" (YAML null values and the empty string question). A common rule: null removes the key, so you can unset a base setting. Another: null is ignored, so a placeholder is harmless. Pick one. Whichever it is, an empty value after a colon is easy to write by accident, so lint for it.
Type conflicts. If the base has a mapping and the override has a scalar for the same key, the override wins in the function above. Often that is a mistake in the override. Decide whether to raise.
\nUnknown keys. A key in the override that is absent from the base may be a typo. A validation step after the merge catches it (validating YAML configuration with pydantic).
\nOrder. Python dictionaries keep insertion order, so the merged dictionary has the base's keys first and then new keys from the override. If order matters in the output file, say so.
\nLayer by folding: functools.reduce(merge, [base, env, local]), or a loop over files sorted by name (10-base.yaml, 20-prod.yaml, 30-local.yaml), each merged onto the result so far. The sorted-by-name directory pattern is easy to reason about and common in tools. In the example in this cluster, the effective config after each file made it clear which file set which value, and printing the intermediate results is a cheap debugging aid (splitting a large YAML config into several files).
Keep the origin of each value if you can. A configuration object that records which file set each leaf lets you answer \"why is the port 8080?\" without guessing. It costs a parallel structure, and it saves hours.
\nIf the merged result is written back to a file that people read, a plain dictionary merge loses comments. Do the merge on ruamel.yaml round-trip objects, which behave like dictionaries; the function works unchanged on them, and comments attached to the base keys survive (preserving comments and order with ruamel yaml). Test that the merge keeps the type of container: the function above copies with deepcopy, and the result is the same class as the input.
yq has a deep-merge operator, *, and an array-appending variant, *+:
yq eval-all 'select(fi==0) * select(fi==1)' base.yaml override.yamlserver:\n host: db2\n port: 5432\n opts: {ssl: true, pool: 20}\nfeatures: [billing]and with *+ the list became [search, export, billing], the same two policies as the Python function. The command is useful in shell scripts and CI (merging YAML files from the command line).
The cluster hub is reading and writing YAML in Python safely. Anchors and merge keys inside one file are in YAML anchors, aliases and merge keys, and environment layering in environment specific YAML config without copy and paste. Python's copy semantics are in the copy module documentation, and Kubernetes' list-merge strategies in its strategic merge patch docs. Merge two snippets visually in the YAML formatter.
\nLists of records have identity. Merging two lists of services by position is wrong. Merge by a key such as name.
The structure is a schema, not data. For Kubernetes-style objects, use the platform's own patch or overlay tool, which knows the fields.
\nYou need to remove things. A merge only adds and replaces. Removal needs an explicit convention, like null meaning delete.
\nThe inputs are untrusted. A merge of attacker-controlled data into trusted defaults can set keys you did not intend. Validate the result against a schema with forbidden extras.
","date_published":"2026-11-04T00:32:00+00:00","tags":["YAML"]},{"id":"https://datahacks.dev/library/getting-mysql-server-running","url":"https://datahacks.dev/library/getting-mysql-server-running","title":"Getting a MySQL Server Running and Usable From Scratch","summary":"A beginner guide to getting a MySQL server running: choose a version, install, secure, connect, create users and fix the first errors.","content_html":"The first hour with a new MySQL server decides a lot of what follows. A server installed carelessly, with a root account that anyone on the network can try, a data directory nobody can find and an application that connects as root, will cause trouble months later, and it will be harder to fix then. The same hour spent well gives a server that is secure by default, documented in a few lines and easy for the next person to understand. This guide walks through that hour: choosing a version, installing, securing, creating accounts, connecting, finding the files and meeting the common errors. It links to the detailed articles in the series for each step.
\nBefore installing anything, answer four questions. Is this a development machine, a shared test server or production? A development database can live in a container and be thrown away, while production needs backups, monitoring and access control from the first day. Who needs to connect, and from where? A single application on the same host is a very different case from several services and analysts across a network. How much data and traffic do you expect? A few gigabytes with light traffic runs on a small virtual machine, and hundreds of gigabytes needs thought about memory and storage. And who will run it? If nobody on the team wants to be responsible for patching, backups and recovery, a managed cloud service may be the better start, with the trade-offs in running PostgreSQL on managed services, which applies in spirit to MySQL services as well. Write the answers in a sentence or two; they will guide the choices below.
\nFor a new project, install MySQL 8.4, the current long term support release. It receives fixes for years without feature changes, which is what a database should do. Innovation releases such as 9.x offer new features sooner and need upgrades every few months, a choice for teams with a testing pipeline. MySQL 8.0 is at or near the end of its supported life and should not host new work. Managed services decide which versions they offer, so check the provider's list and its retirement dates before choosing. MariaDB is a separate product with differences that this series does not assume. The reasoning and the support calendar are in which MySQL version to choose for a new project.
\nThere are two good routes. A native package installs the server as a service on the host, managed by the operating system. On macOS use Homebrew or the Oracle installer, on Ubuntu and Debian the MySQL APT repository, on Red Hat family systems the MySQL Yum repository, on Windows the MSI installer. Each puts the data in a conventional directory and registers the service, and the platform differences are covered in installing MySQL 8.4 on macOS, Linux and Windows. The second route is a container: the official mysql:8.4 image with a named volume for the data directory runs identically on every developer's machine and disappears cleanly when you are done, as shown in running MySQL in Docker with a data volume. Containers shine for development and testing. For a server that lives for years, a native install or a managed service is simpler to operate, unless your organization already runs containers well.
A fresh server needs five minutes of hardening, and it is much easier now than after an application depends on it. Set a long, generated password for the root account, using ALTER USER 'root'@'localhost' IDENTIFIED BY '...'. Run mysql_secure_installation, which removes anonymous accounts and the test database and disallows remote root login, covered in removing anonymous users and test databases. Keep the server from listening on the public internet: bind to localhost or a private address, and put a firewall in front. If remote clients must connect, require TLS and restrict the accounts by host. Keep the server patched with minor updates. These steps are cheap, and skipping them is how databases end up in breach reports.
Create the database with utf8mb4 and a clear name. Then create separate accounts for separate jobs: one for the application with SELECT, INSERT, UPDATE, DELETE on its database, one for schema migrations with the data definition privileges, one read-only for reporting, and a personal account for each administrator. Each account is a user and a host pattern, and the host pattern should be as narrow as the network allows. Never let an application use root. The statements, the role feature in MySQL 8 and the verification steps are in creating a database and a least privilege user in MySQL. Keep the statements in version control so that every environment is created the same way, and use generated passwords stored in a secret manager.
The mysql client is the baseline tool. It connects through a Unix socket when the host is localhost and through TCP for any other name, and the two paths match different accounts and fail in different ways. Put repeated settings in an option file or a login path, keep passwords off the command line, request TLS with --ssl-mode when crossing a network and use an SSH tunnel for private servers. The options and the habits are in connecting to MySQL from the command line. Most \"cannot connect\" problems resolve by reading the error number: 2002 is a socket problem, 2003 a network problem, 1045 a credentials problem.
Before you spend hours in a production console, configure the client. --safe-updates makes the server refuse UPDATE and DELETE statements without a key-based WHERE, and caps accidental huge selects. A pager keeps long results readable, \\G prints wide rows vertically, and a prompt that shows the host and database tells you where you are before you press Enter. Put them in the [mysql] section of your option file, as described in using mysql client options like safe updates and pager. Wrap manual data changes in transactions and check the row count before committing. These habits cost nothing and prevent the incidents that are remembered for years.
Most mysteries about a MySQL server are answered by knowing where its files are. The server tells you: SELECT @@datadir, @@socket, @@log_error. The configuration comes from layered option files, read in a fixed order with later settings winning, plus the mysqld-auto.cnf file written by SET PERSIST. The data directory holds the InnoDB tablespaces, redo and undo logs, one folder per database and the binary logs if enabled. Defaults differ between Debian, Red Hat, macOS, Windows and Docker. Never copy or delete files in the data directory while the server runs. The layout and the order of option files are explained in where MySQL keeps its config and data files.
Three errors bring most newcomers to a search engine. ERROR 1045 Access denied for user 'root'@'localhost' means the server was reached and refused the credentials: check the password, whether root uses socket authentication, which host the server saw and whether the client supports the authentication plugin. The checklist is in fixing access denied for user root at localhost. ERROR 2002 Can't connect to local MySQL server through socket means the client looked for a socket file that is missing or unreachable: check that the server runs, find the real socket path and use TCP with 127.0.0.1 if needed, see fixing can not connect to local MySQL server through socket. And the lost root password, which you reset with an init file as described in resetting a lost MySQL root password. Know these three and you will recover from the majority of first-day problems.
A few server settings deserve a decision early, because changing them later is painful. Use utf8mb4 as the server and database character set, and a collation that matches your comparison needs. Decide how time is stored: set the server time zone deliberately (UTC is the safest default for servers) and choose between DATETIME and TIMESTAMP with the trade-offs in DATETIME versus TIMESTAMP in MySQL. Keep the default strict sql_mode, which makes the server reject invalid data instead of silently truncating it; relaxing it to satisfy a sloppy application only hides bugs. These settings go in the configuration file, under version control, with a comment on why each is set.
A server without a tested backup is a risk that grows with every row. Decide the method early: logical dumps with mysqldump or the MySQL Shell utilities for small and medium databases, physical backups with Percona XtraBackup for large ones, binary logs for point-in-time recovery. Schedule them, send them off the machine, and restore one into a scratch server to prove that it works. The details are in the backup series, starting with MySQL backups you can count on. Even on a development machine, a script that dumps the schema and seed data into the repository saves hours when the volume is lost.
Even a small server benefits from a few checks. Watch that it is up and accepting connections, that disk space is not running out, that the error log does not fill with warnings, that the number of connections stays below max_connections and that backups ran. Later you will add query performance, replication lag and buffer pool statistics, covered in the tuning and performance series. A cron job that mails on failure is better than nothing, and a metrics stack is better still. Enable the slow query log with a sensible threshold early, see turning on the slow query log safely, because it costs little and gives you evidence when someone says the database is slow.
When you set up a server, run through a short list. Version chosen and recorded. Installed from a supported source. Root password changed and anonymous accounts gone. The server bound to the right interface, with a firewall. Database created with utf8mb4. Application, migration and reporting accounts created with narrow grants and host patterns. TLS configured for remote connections. Client defaults set for safety. Paths of the configuration, data and logs written down. Time zone and sql_mode set deliberately. Backup job running, and one restore tested. Slow query log on. Monitoring and an alert for disk space and for the service being down. A runbook page with the commands for the three first errors. Keep the checklist in the repository and update it after each setup, and the next server takes ten minutes instead of an afternoon.
The most common serious mistake with a new MySQL server is leaving it reachable from places it should not be. The server listens on port 3306, and bind_address decides which interfaces accept connections: 127.0.0.1 means only the local machine, * means all. For a single-host setup, keep it on loopback. For a server shared with other machines, bind to the private address and open the firewall for the specific clients only. Cloud security groups, ufw, firewalld and the Windows firewall all do this, and a rule that allows 0.0.0.0/0 on 3306 is the line to look for in any review. Docker's published ports bypass some host firewalls, so publish to 127.0.0.1 explicitly. If people need access from laptops, give them an SSH tunnel or a VPN, not an open port. Scanners find open MySQL ports within hours, and a weak password on one is enough.
The default configuration is conservative. A few settings are worth setting at the start. Set innodb_buffer_pool_size to a value that fits your memory, about half to three quarters of RAM on a dedicated database host, less when the machine runs other services; the reasoning is in how big should innodb_buffer_pool_size be. Set max_connections to a number you can afford in memory, not an arbitrary large value. Turn on the slow query log. Set character_set_server to utf8mb4. Set the time zone. Enable binary logging if you will need point-in-time recovery or replication, and set an expiry for old binary logs, otherwise they fill the disk. Leave the rest alone until you have measurements, and write each change with its reason as a comment in the configuration file.
Time handling trips up many beginners. MySQL stores TIMESTAMP values in UTC and converts them to the session time zone on the way in and out, while DATETIME stores what you give it with no conversion. The server has a global time zone setting, and each session can override it, so the same stored value can display differently to two clients. Set the server to UTC, load the time zone tables if you want named zones (mysql_tzinfo_to_sql on Linux), and have applications set the session zone explicitly. Keep the operating system clock synchronized with NTP, since replication, certificates and log analysis depend on it. The choice of column type has consequences for ranges and conversions, covered in DATETIME versus TIMESTAMP in MySQL.
Treat updates as routine work. Minor releases, such as 8.4.2 to 8.4.3, contain security and bug fixes and are safe to apply with a restart. Subscribe to Oracle's security advisories, which come quarterly, and apply them within a defined time. Package managers do most of the work, and containers need a new image tag. Test the update on a copy first if the server is important, and take a backup before. Major changes, such as the move to the next LTS release, need more care: read the upgrade notes, test the application, plan the downtime and keep a rollback path. Replicas are upgraded before the source. A calendar entry for the end-of-support date of your version prevents the surprise of running unpatched software. The version policy is explained in which MySQL version to choose for a new project.
\nLogs are how you learn what a server did. The error log is on by default and is the first place to look for any startup problem or crash. The slow query log, off by default, records statements slower than long_query_time; a threshold of one second or less with log_slow_extra gives useful detail. The general log records every statement and is only for short debugging, since it is large and can contain passwords. The binary log records changes for recovery and replication. Rotate logs with logrotate or the server's own options, watch their size, and protect them, as they can contain sensitive data. Where logs go in containers, on managed services and under systemd differs, so find out early.
A development database and a production database should be as similar as possible in version, character set, SQL mode and schema, and as different as possible in size, access and data. Developers need a quick way to create a fresh copy: a container, a dump of the schema and generated sample data, not a copy of production data, which brings privacy risk. Production needs access control, backups, monitoring and a change process. Keep the configuration files in the same repository and differ only in a small set of values. A common failure is the application that works on a laptop with permissive settings and fails in production with strict ones; run the test suite against a server with the production sql_mode and character set.
Applications reach MySQL through drivers and frameworks that bring their own defaults. Check the connection string options: the character set, the time zone, whether TLS is requested, the connection pool size and the timeouts. Check that the driver supports the authentication plugin your server uses; recent versions of the mainstream drivers support caching_sha2_password. Set the pool size from the server's connection budget and not from the framework's default, and set a connection timeout so that an unavailable server fails fast. Prepared statements and parameter binding in the driver, not string concatenation, are the first defense against injection, covered in preventing SQL injection in MySQL applications. Test failover behavior if you run replicas: does the application reconnect after the server restarts?
When something does not work, move from the outside in. Is the host reachable (ping, DNS)? Is the port open (nc -vz host 3306)? Is the server running (systemctl status, mysqladmin ping)? Does the error log show a problem? Can you connect locally over the socket? Over TCP? With which account? What do SHOW GRANTS and mysql.user say? Is the plugin compatible? Is TLS required and configured? Each question narrows the layer, and the error number says where to begin: 2002 and 2003 are connectivity, 1045 is authentication, 1044 and 1142 are privileges, 1049 is an unknown database. Write the flow in your runbook with the commands for your environment. A calm checklist beats a panicked guess at 3 a.m.
A database server needs an owner: a named person or team who decides on upgrades, grants access, watches alerts and answers questions. Write down in one page where the server lives, how it was installed, which version it runs, where the configuration is, who has access, how backups work and where, how to restore and how to reach the owner. Review the page twice a year. The most useful documentation is short and exact: commands that were tested, not descriptions of intentions. When someone leaves the team, the page and the secret manager are what keep the server operable. Treat undocumented servers as a risk to be fixed, whatever their age.
\nMySQL is more sensitive to disk than to almost anything else. Use SSD or NVMe storage for the data directory, and keep the redo log and binary logs on storage with predictable latency. A busy server on a slow disk spends its time waiting, and no setting repairs that. Leave headroom: a data volume that is 90 percent full leaves no space for index builds, temporary tables or ALTER TABLE operations that copy a table. Memory comes next, since a buffer pool that holds the working set turns most reads into memory hits. CPU matters for concurrent queries, and a few fast cores serve many small workloads better than many slow ones. In virtual machines and clouds, check the disk's IOPS and throughput limits, because credits and burst allowances can run out under sustained load. Plan growth: measure the size of the data directory monthly, and extend storage well before it fills.
After the first hour, the work continues for a few days. Add a monitoring agent or a metrics exporter, and dashboards for connections, queries per second, the buffer pool and replication. Set alerts for disk space, service availability and failed backups. Write a runbook entry for how to restore. Run a restore into a scratch server and time it. Review the user list and the grants. Check the slow query log after a day of real traffic and fix the worst query. Look at the error log for warnings that appear every minute and silence them by fixing the cause. Put the configuration in version control. Schedule the next review in a month. Small servers that receive this attention in the first week tend to stay healthy for years, and those that do not tend to be rescued in a hurry after an incident.
\nIf your team lacks the time or the inclination to run a database, a managed MySQL service removes the installation, patching, backups and failover from your list and adds a bill. You still do the account design, the schema, the queries and the monitoring, and you lose some control: no host access, a restricted set of settings and privileges, and a provider-set upgrade schedule. For a small team building a product, that trade is usually right. For a team with operations experience and specific needs, a self-run server is fine. The decision is reversible with a dump and restore, so it need not be agonized over; pick one, document it and review in a year.
\nThe articles in this series cover each step: installing MySQL 8.4 on macOS, Linux and Windows, running MySQL in Docker with a data volume, creating a database and a least privilege user in MySQL, connecting to MySQL from the command line, fixing access denied for user root at localhost, fixing can not connect to local MySQL server through socket, resetting a lost MySQL root password, where MySQL keeps its config and data files, which MySQL version to choose for a new project and using mysql client options like safe updates and pager. The authority is the MySQL 8.4 reference manual and the installing and upgrading chapter, and the next topics are data types in picking MySQL data types with care and query writing in writing MySQL 8 queries that use modern features.
","date_published":"2026-11-03T15:33:00+00:00","tags":["MySQL"]},{"id":"https://datahacks.dev/library/json-allows-parsers-quietly-accept","url":"https://datahacks.dev/library/json-allows-parsers-quietly-accept","title":"What JSON Allows and What Parsers Quietly Accept in Practice","summary":"The JSON grammar fits on a page, and parsers still disagree on trailing commas, NaN, duplicate keys, big numbers and more. A 26-case test across five parsers.","content_html":"JSON is the one data format that every developer thinks they understand. It has six kinds of value, a grammar that fits on the back of a business card and a name that suggests it is just JavaScript. Then a config file with a trailing comma works in one tool and fails in another, a service returns an ID that is off by three in the last digit, and two systems read the same document and disagree about what it says. The format is simple. The edge cases, the places where the specification says \"may\" or says nothing, are where parsers go their own ways, and most real JSON bugs live there. This guide lays out what the grammar allows, then shows what five common parsers do with the inputs at its edges, based on a 26-case experiment, and ends with how to write and read JSON so that those differences do not matter. The tooling around the format, including the command-line side, is surveyed in JSON parse errors and how to read them, and the detailed cheat sheet of test inputs is in a cheat sheet of JSON edge cases to throw at a parser.
\nJSON is defined by RFC 8259 and, in nearly identical terms, by ECMA-404. A value is one of six things: an object, an array, a string, a number, true or false, or null. That is the whole type system; there is no date, no binary, no integer as distinct from a float, no comment and no way to say \"undefined\".
An object is braces around zero or more \"name\": value pairs separated by commas, and the name must be a string in double quotes. An array is brackets around zero or more values separated by commas. A string is double quotes around characters, with the escapes \\\", \\\\, \\/, \\b, \\f, \\n, \\r, \\t and \\uXXXX, and with raw control characters (below U+0020) forbidden: a literal tab or newline inside a string is not allowed. A number is an optional minus, an integer part with no leading zeros (0 alone, or a digit 1 to 9 followed by digits), an optional fraction that needs digits on both sides of the point, and an optional exponent. Whitespace, meaning space, tab, newline and carriage return, may appear between tokens.
Since RFC 7159, a document can be any value, so 42 and \"hello\" are valid JSON texts on their own, not only objects and arrays. The older RFC 4627 allowed only the two containers at the top. Nothing in the grammar allows trailing commas, comments, single-quoted strings, unquoted names, NaN, Infinity, hexadecimal, a leading plus sign, a number like .5 or 1. or a second document after the first. A parser that accepts any of those is accepting more than the format, and that is where the tour begins.
The RFC is also explicit that parsers have latitude. It says that an implementation may set limits on the size of texts, the depth of nesting, the range and precision of numbers and the length of strings, and that a parser may accept non-JSON forms. So \"not valid JSON\" is not a property of the bytes alone: it is a statement about a parser, and the next section shows how much parsers vary.
\nI wrote 26 inputs, each a small file with exact bytes, and asked five parsers whether they accept it. The parsers were Python's json.loads on text, Node's JSON.parse, PHP's json_decode, jq 1.6 and DuckDB's json_valid. Here are the results for the inputs that matter most, with ok for accepted and ERR for rejected:
| Input | Python | Node | PHP | jq 1.6 | DuckDB |
|---|---|---|---|---|---|
{\"a\": 1,} trailing comma in object | ERR | ERR | ERR | ERR | ok |
[1, 2,] trailing comma in array | ERR | ERR | ERR | ERR | ok |
{'a': 1} single quotes | ERR | ERR | ERR | ERR | ERR |
{a: 1} unquoted key | ERR | ERR | ERR | ERR | ERR |
// note then an object | ERR | ERR | ERR | ERR | ERR |
[NaN] | ok | ERR | ERR | ok | ok |
[Infinity] | ok | ERR | ERR | ok | ok |
{\"a\": 1, \"a\": 2} duplicate keys | ok | ok | ok | ok | ok |
[01] leading zero | ERR | ERR | ERR | ok | ERR |
[+1] leading plus | ERR | ERR | ERR | ok | ERR |
[.5] and [1.] | ERR | ERR | ERR | ok | ERR |
[\"a<TAB>b\"] raw tab in a string | ERR | ERR | ERR | ERR | ERR |
[\"\\ud800\"] lone surrogate escape | ok | ok | ERR | ERR | ERR |
| UTF-8 BOM before the object | ERR | ERR | ERR | ok | ERR |
42 top-level scalar | ok | ok | ok | ok | ok |
{\"a\": 1}{\"b\": 2} two documents | ERR | ERR | ERR | ok | ERR |
| empty input | ERR | ERR | ERR | ok | ERR |
| 5,000 levels of nesting | ok | ok | ERR | ERR | ok |
The agreement is on the clear cases: single quotes, unquoted names, comments, raw control characters, hex numbers and bad escapes were rejected everywhere, and top-level scalars, huge exponents and big integers were accepted everywhere. Everything else splits the field.
\nSome details need caveats. For Python, the BOM case depends on the input type: json.loads on a string beginning with the BOM character raises an error (\"Unexpected UTF-8 BOM\"), and on bytes it detects the BOM and accepts the document, which is why the same file can parse in a script that reads bytes and fail in one that reads text. jq accepted the two-documents input because it is a stream processor that reads values one after another, and accepted empty input because it printed nothing. DuckDB's json_valid is strict about numbers and quotes and lenient about commas, NaN and Infinity. PHP rejected the deeply nested input because json_decode defaults to a maximum depth of 512, which is a parameter and not a failure of the grammar.
Acceptance is half of it. The values that come out differ too. For the same four inputs, Python, Node, PHP and jq produced:
\n| Input | Python | Node | PHP | jq 1.6 |
|---|---|---|---|---|
{\"a\": 1, \"a\": 2} | {'a': 2} | {\"a\":2} | {\"a\":2} | {\"a\":2} |
[1e400] | [inf] | [null] after stringify | overflow, not encodable | [1.7976931348623157e+308] |
[12345678901234567890123] | exact integer | 1.2345678901234568e+22 | 1.2345678901234568e+22 | 12345678901234568000000 |
[9007199254740993] | 9007199254740993 | 9007199254740992 | 9007199254740993 | 9007199254740992 |
Duplicate keys resolve to the last value in all four, a convention and not a rule: the RFC says names within an object \"SHOULD be unique\" and that behavior with duplicates is unpredictable. DuckDB's JSON type keeps the original text, duplicates included, until you extract a value. The overflowing exponent becomes infinity in Python, null after Node serializes it back and the largest double in jq. The 23-digit integer is exact in Python, which has arbitrary-precision integers, and a rounded float everywhere else.
The last row is the one that bites. 9007199254740993 is 2 to the 53rd plus one, the first integer that a 64-bit float cannot represent. Node and jq read it as the float and print 9007199254740992, off by one, and the reader has no way to know. PHP kept the exact 64-bit integer, and Python kept it as an arbitrary-precision one. If that number is an order ID, the same document identifies two different orders in two services (how JSON handles big numbers and why IDs lose digits has the fixes).
JSON was specified after it was in use. Douglas Crockford described it as the subset of JavaScript literals that was safe to exchange, and implementations grew from eval and from hand-written parsers that accepted what their authors thought was reasonable. Standards then wrote down what the strictest reading allowed, and left a lot to implementations. Every deviation in the table has a reason: Python allows NaN and Infinity because they are Python floats and json.dumps writes them by default; jq 1.6 was lenient about numbers because its parser accepts anything that looks like a numeric literal; DuckDB reads trailing commas because its JSON reader, built for analytics on messy files, is deliberately forgiving.
The consequence is that a document can be valid for the system that wrote it and invalid for the one that reads it. A Python service that emits NaN for a missing measurement produces output that a JavaScript client cannot parse (is NaN or Infinity valid in JSON). A config file with a trailing comma that a lenient tool accepts breaks the first strict tool in the pipeline (trailing commas in JSON and why parsers reject them).
The table is useful as a way to choose what to write. Take the rows where all five parsers agree on \"reject\": single quotes, unquoted names, comments, raw control characters, hexadecimal. Writing any of those is a guaranteed failure everywhere, and the fix is to stop. Take the rows where they split on rejection, trailing commas, NaN, leading zeros, a BOM and lone surrogates. Those are the risky ones: they work in your tests, because your parser tolerates them, and fail in production, because the consumer is stricter. The safe response is to treat the strictest parser's reading as the standard. Take the rows where all five accept but the values differ, duplicate keys and big integers: nothing fails and the data is quietly different. Those are the most dangerous, since no error ever tells you.
That three-way sort, fails everywhere, fails somewhere, works everywhere with different meaning, is how I would triage any new JSON producer or consumer. Run the cases, put each in a bucket and fix the second and third first, because the first announces itself.
\nThe RFC says that JSON exchanged between systems that are not part of a closed ecosystem must be encoded in UTF-8, and that implementations must not add a byte order mark, though they may ignore one when reading. That is the specification, and it explains the BOM row in the table: Python on bytes ignores it, and on text raises, Node and PHP reject it, jq and others skip it. A producer that writes a BOM is violating a MUST NOT, and a consumer that rejects it is within its rights, so the practical answer is to never write one and to strip one defensively when you read files that came from Windows editors.
\nEscapes make the second source of trouble. \\u00e9 and the raw character é are the same string. A character outside the Basic Multilingual Plane, such as the grinning face U+1F600, is written in an escaped form as a surrogate pair, \\ud83d\\ude00, two escapes standing for one character. Python's json.dumps writes that form by default (ensure_ascii=True) and the raw character with ensure_ascii=False; both parse back to the same string. The problem is a surrogate escape on its own, \\ud800 with no partner. The grammar allows it, since it is a valid escape sequence, and it names no character. Python and Node accept it, PHP and jq reject it, and a library that then tries to encode the string as UTF-8 will fail or write a replacement character. Reject lone surrogates at the boundary (unicode escapes and surrogate pairs in JSON).
String length shows the same split in another form. The grinning face is one character, four bytes in UTF-8 and two units in UTF-16. Python's len returned 1, jq's length returned 1, and JavaScript's .length returned 2, because JavaScript counts UTF-16 code units. A validation rule of \"at most 20 characters\" therefore means different things in a Python service and in a JavaScript form, and an emoji-heavy name fits in one and not the other. Count what the rule means and say which unit in the specification.
The number grammar says nothing about size or precision. 1, 1.0 and 1e0 are all legal, and a parser decides what to do with them. Python keeps the distinction: json.loads(\"1.0\") is a float and json.loads(\"1\") is an integer, and json.dumps(-0.0) writes -0.0. JavaScript has only one number type, so JSON.parse(\"1.0\") and JSON.parse(\"1\") are both 1, and JSON.stringify(-0) writes 0. A round trip through JavaScript therefore turns 1.0 into 1, which is harmless unless a downstream system reads 1.0 as a decimal with a scale. The size problem was shown above: any integer above 2 to the 53rd is at risk in a parser that uses doubles.
Treat numbers that must be exact, money, identifiers, counters beyond 9 quadrillion, as strings, or as integers in a documented range, and test the path through every parser that reads them.
\nYou can make Python's parser as strict as the specification with two hooks, one for the constants and one for object pairs, and a defensive BOM strip:
\nimport json\n\ndef reject_constant(name):\n raise ValueError(f\"non-standard number {name} is not allowed\")\n\ndef reject_duplicates(pairs):\n seen = {}\n for key, value in pairs:\n if key in seen:\n raise ValueError(f\"duplicate key {key!r}\")\n seen[key] = value\n return seen\n\ndef strict_loads(text):\n if text.startswith(\"\\ufeff\"):\n text = text[1:]\n return json.loads(text, parse_constant=reject_constant, object_pairs_hook=reject_duplicates)Run on five inputs:
\n'{\"a\": 1, \"b\": [true, null]}' -> {'a': 1, 'b': [True, None]}\n'[NaN]' -> rejected: non-standard number NaN is not allowed\n'{\"a\": 1, \"a\": 2}' -> rejected: duplicate key 'a'\n'{\"a\": 1,}' -> rejected: Illegal trailing comma before end of object: line 1 column 8 (char 7)\n'\\ufeff{\"ok\": true}' -> {'ok': True}The valid document passes, NaN and the duplicate are rejected by the hooks, the trailing comma was already rejected by the parser (Python 3.13 words the error as \"Illegal trailing comma before end of object\"), and the BOM-prefixed document passes because the function strips it. About fifteen lines turn a lenient default into the strict contract. The equivalent hooks exist in other languages: a reviver or a custom parser in JavaScript, JSON_THROW_ON_ERROR and a depth argument in PHP, and a schema validator in front of everything.
Two JSON texts can be different byte for byte and the same document. Whitespace between tokens is free, and the order of keys in an object has no meaning in the specification. In Python, json.dumps({\"b\": 1, \"a\": [1, 2]}) gives {\"b\": 1, \"a\": [1, 2]}, and the compact form of the same data with sorted keys is {\"a\":[1,2],\"b\":1}: the two parse to equal values and compare unequal as text. That matters whenever you hash, sign or diff JSON. A hash of the text changes if a library changes its spacing or key order, so a signature needs a canonical form. RFC 8785, the JSON Canonicalization Scheme, defines one (sorted keys, no extra whitespace, a specified number format), and some libraries implement it. For diffs, sort the keys and indent consistently before comparing, or use a tool that compares parsed values.
For what you send: UTF-8, no BOM, unique keys, finite numbers only, IDs and exact decimals as strings, timestamps as ISO 8601 strings with an offset (JSON dates and the lack of a date type), a declared content type of application/json and nothing before or after the document. For what you receive: a size limit before parsing, a depth limit, a strict parser configuration as above, schema validation after parsing, an explicit decision about unknown fields (ignore or reject) and the same error shape for every kind of failure, so that clients can tell malformed JSON from valid JSON that violates the rules. Document both, with an example payload that a client can paste, and keep a conformance test with the awkward inputs next to the code that parses them.
It helps to know why the format is both tiny and untidy. JSON started as a way to send data to a browser: the objects and arrays were already valid JavaScript, so a page could read a response with eval. Douglas Crockford named and described the subset, and the first spec, RFC 4627 in 2006, was a few pages. Early parsers were small and written by people who wanted them to work on the data they had, so they accepted a little more than the grammar. When languages other than JavaScript adopted the format, their authors made their own choices about what to do with NaN, with integers beyond the float range and with repeated names, and each choice became behavior that existing programs depended on. Later standards could tighten the text and could not change a million deployed parsers.
The effect is that strictness is a property of a particular parser at a particular version, and almost never an accident of the format. A parser that is lenient today has users who rely on it, and a parser that is strict is easier to reason about and breaks files that used to work. Neither is wrong, and neither is safe to assume.
\nOne more edge is size. The grammar allows a document of any length, and most parsers build the whole structure in memory, so a 2 GB array of records needs several gigabytes of RAM to read, and a parse that fails at the last byte has wasted all of it. For large collections, the common alternative is one JSON value per line, called JSON Lines or newline-delimited JSON, where each line is a complete document. A reader handles a line at a time, a writer can append, and a damaged line affects one record and not the file. DuckDB reads it with read_json and read_ndjson, jq processes it as a stream, and Python's loop over lines with json.loads does the same in a few lines. When you control the format of a large export, choose JSON Lines over one big array (converting CSV to JSON Lines for streaming shows the conversion). When you receive a big array, use a streaming parser, such as ijson in Python or jq --stream, instead of loading it.
Choosing JSON for data exchange is a choice among formats, and each has its own edges. CSV is flat, typeless and universal in spreadsheets, and has no standard for nesting. XML has schemas, namespaces, comments and attributes, and a verbose syntax with its own parser pitfalls. YAML is a superset in spirit that allows comments, anchors and many ways to write a value, with the implicit typing that turns no into a boolean and 1.10 into a number, which is why it is a poor format for data from outside. TOML and INI suit small configuration files. JSON sits in the middle: typed enough for APIs, small enough to parse everywhere, with no comments and no date. If you need to compare, look at what each does to your awkward values: a long ID, a date, a missing value, a text with a quote and a newline. The format that preserves them with the least ceremony is the right one for that data.
Most of the trouble people report is not a spec question. It is a file that is nearly JSON: a trailing comma left by a hand edit, single quotes from a Python repr, a JavaScript object literal pasted into a file, a truncated response, an HTML error page where JSON was expected, a BOM from an editor. Each has a specific diagnosis and a specific fix, and the error message points at it when you know how to read it. The next articles in this series go through them: reading parse error messages, single quotes, trailing commas, comments and the others. Repair is a last resort: where you can, fix the producer so that the documents are valid, and where you cannot, use a purpose-built lenient parser such as JSON5 for files that you know come from humans, rather than regular expressions that guess.
Disagreement between parsers is not only an annoyance. If two components read the same JSON and take different values from it, an attacker can write a document that one component validates and the other acts on. Duplicate keys are the classic case: {\"role\": \"user\", \"role\": \"admin\"} validated by a parser that keeps the first value and executed by one that keeps the last. A lone surrogate, a huge number or a differently handled BOM can do similar work. Nicolas Seriot's survey, Parsing JSON is a Minefield, tested dozens of parsers against hundreds of inputs and documented how much they vary, which is the larger version of the experiment here.
Defenses are practical. Parse once, at the boundary, and pass the parsed structure inward, never the text. Reject duplicate keys where you can (in Python through object_pairs_hook, shown in duplicate keys in a JSON object and which one wins). Validate against a schema after parsing. And limit size and depth, since a deeply nested document can exhaust a parser's stack.
The advice splits by direction. When you produce JSON, produce the strict, boring subset that every parser agrees on. RFC 7493, the I-JSON profile, is a written version of it: object member names unique, numbers within the range of an IEEE double, no non-finite numbers, UTF-8 and no unpaired surrogates, and dates and binary data as strings in agreed encodings. Write IDs above 2 to the 53rd as strings, write times as ISO 8601 strings, omit fields instead of writing NaN, and write documents with a library and never by hand-concatenating strings.
When you consume JSON, be as strict as your needs allow, and say so. Use the parser's strict mode and turn off non-standard extensions (allow_nan=False in Python's json.loads with a parse_constant that raises). Handle the BOM on purpose, since files saved by Windows tools often have one. Decide what to do about duplicates. Limit depth and size. And read errors properly: the message tells you the position and the expectation, and fixing JSON parse errors by reading the message properly shows how to use them.
If JSON from outside enters your system, test your parser with the awkward inputs, because its behavior is the one that matters. Keep the 26 inputs, or a larger set such as the JSONTestSuite, in a folder, run them through the code path you use in production and write down the accept and reject list. Check it again when you upgrade the library or the runtime. A cheat sheet of JSON edge cases to throw at a parser has the set, the scripts that produced the table above and the expected results for each case. A parser's behavior at these edges is stable for years and changes without much notice when a major version arrives, so a test that fails on upgrade is useful information.
\nStrictness is for data exchanged between programs. For files that people edit, configuration, settings, fixtures, a format with comments and trailing commas is kinder, and the right move is to choose one openly instead of bending JSON: JSONC for editor settings, JSON5 for human-edited config, YAML or TOML where those fit (JSON versus JSON5 versus JSONC and when to use each compares them). A file with comments in a .json file works in the tool that expects it and is a bug waiting in any other. The authority for the grammar is RFC 8259, with the interoperability profile in RFC 7493, and Python's behavior is documented in the json module reference.
\"Export, edit in a spreadsheet, import\" is a workflow users expect to work. They download their customers, fix forty cells, upload the file and assume that the other 3,960 rows come back as they were. Whether they do is a property you can test, and the test most people write is the wrong one. They export twice and compare the files, find them identical and conclude the feature is safe. This article builds the right test, shows with a real run how the wrong one passes while the data changes and goes through the places where a CSV round trip loses information. It is one of the contracts discussed in CSV import and export features in applications, and the NULL question it exposes is the same one that handling null and empty strings in PostgreSQL examines from the database side.
\nA source table with four rows, each testing a different hazard:
\n| id | name | note | active | balance |
|---|---|---|---|---|
| 1 | Acme Ltd | NULL | 1 | 120.50 |
| 2 | Beta, Corp | empty string | 0 | 0.00 |
| 3 | Cy \"Foods\" | two newline lines | 1 | 15.00 |
| 4 | Delta (padded) | x | 1 | 007.10 |
The steps: export the table to a CSV with fputcsv, import that file into a second, identical table, and export the second table. Then compare. The CSV that comes out of the first export is:
id,name,note,active,balance\n1,\"Acme Ltd\",,1,120.50\n2,\"Beta, Corp\",,0,0.00\n3,\"Cy \"\"Foods\"\"\",\"two\nlines\",1,15.00\n4,\" Delta \",x,1,007.10Look at rows 1 and 2. The note is empty in both, and the file does not say which was NULL and which was an empty string. (PHP quotes Acme Ltd because it contains a space, a quirk of fputcsv, harmless but worth knowing when you compare against files from other writers.) The comma, the quotes, the line break, the padding and the leading zero in 007.10 are all preserved: quoting and doubling handle the first three, and the last two are plain text in a text column.
Compare the first export with the second:
\nsecond export identical to first: trueThe files are byte for byte the same, and the test passes. Now compare the tables, row by row, looking at how each note was stored before and after:
\nrows that differ after the round trip: 1\n id 1 note before NULL after ''\n id 2 note before '' after ''Row 1's note was NULL in the source and is an empty string in the copy. The round trip changed the data, and the second export, which writes NULL and empty string identically, could not show it. Any check that goes through the file format is blind to a difference that the format cannot express. That is the general lesson: compare the data in its native form, on both sides of the round trip, not the files in the middle.
\nThe other rows came through intact, including the one with the multi-line note and the quotes. In a table where the distinction between NULL and empty matters (a nullable foreign key as text, an optional field where \"unset\" differs from \"set to nothing\"), the loss is real. Where it does not matter, it is a documented simplification.
\nThere are exactly two readings of an empty cell, and each loses something. Read an empty cell as an empty string, and a NULL becomes '', which is what the test above did to row 1. Read it as NULL, and rows 1 and 2 both come back as NULL:
--- with empty -> NULL on import\n id 1 note before NULL after NULL\n id 2 note before '' after NULLRow 1 is now right and row 2 is wrong. Neither policy round-trips both, because the information is not in the file. The ways out are the following.
\nDecide that the distinction does not exist at the application level: store NULL for \"nothing\" everywhere, and normalize on write, so the table can never hold an empty string in that column. Then reading empty as NULL is lossless. This is the simplest and best answer for most text fields.
\nOr make the file carry the difference. RFC 4180 has no rule for it, but a common convention is that an unquoted empty field is NULL and a quoted empty field (\"\") is an empty string, which PostgreSQL's COPY ... CSV follows (PostgreSQL CSV versus text format in COPY). The writer must then force quotes on empty strings, which PHP's fputcsv does not do, and the reader must distinguish quoted from unquoted, which many parsers hide. Or use an explicit marker such as \\N or NULL, documented, with the risk that a real value NULL collides.
Or accept the loss and say so: \"Empty cells are imported as empty text.\" That is honest, and it is better than leaving it to chance.
\nThe table above had four rows, and several other hazards can be tested the same way. Types: numbers written with trailing zeros (120.50) come back as 120.5 if the import parses them to floats and the export formats them again; keep decimals as decimals or as text and format explicitly. Dates and time zones: a timestamp exported in local time and imported as UTC moves by the offset; write ISO 8601 with an offset. Booleans: 1 and 0 out, true and false in, or the reverse; accept both on import and pick one for export. Whitespace: padding survived in the test, and many importers trim, which changes the data and deserves a rule of its own. Encoding: the writer's UTF-8 and the reader's default code page. Line endings: embedded line breaks may turn into \\r\\n after passing through a Windows tool. Order: unless the export sorts, the order of rows is unspecified, so a comparison by position fails and one by key does not.
Spreadsheets add their own. Excel opening the exported file and saving it can strip leading zeros from 007.10 if the cell was typed as a number, round long numbers to 15 digits, convert 1-2 to a date and rewrite line endings (opening CSV files in Excel and Sheets without damage). A round trip through a spreadsheet is a different test from a round trip through code, and it is the one your users do. Test both: write a file, open it in the spreadsheet your users use, save as CSV and import it.
The test belongs in your suite, and it is short. Load fixture rows that hit each hazard into the source table. Export. Import the file into an empty target table. Compare the two tables, by key, column by column, with an equality that treats NULL as unequal to the empty string, and fails with the row and column of the first difference. Then assert the second export equals the first as a cheap secondary check, knowing it is secondary.
\nThe fixture matters more than the code. Include a NULL, an empty string, a value with a comma, a quote, a line break, leading and trailing spaces, a leading zero, a long number, a date at midnight, a negative decimal, a non-ASCII name and a value that starts with =. If your system cannot round-trip one of them, the test should say so, and you decide whether to fix it or document it. Keep the fixture in the repository, add a row for every bug a user reports and run the test in CI. A round-trip test that checked only the happy path is a test that passes on the day before the incident.
Some data cannot go through CSV unchanged. Binary values, nested structures, very large numbers and values with mixed types in one column all need an encoding that the file does not give you. A relation with line items does not fit in one flat table without a convention. For those, the honest design says which columns are exportable and which are not, and offers a different format (JSON, a zip of files) for the whole record. If the feature promises \"edit and re-import\", restrict it to the columns that survive, and state the limits in the template and the interface, since a user who loses a nuance in a note field will blame the tool and not the format.
\nAlso be clear about the direction that is allowed to be lossy: an export for reading, such as a report, can drop information and format values for humans, and an export meant to be re-imported cannot. Build them as separate outputs, from the same column definitions, with names that tell users which is which. The behavior of the PHP functions is documented in the fgetcsv and fputcsv pages, and the format in RFC 4180.
","date_published":"2026-11-02T15:11:00+00:00","tags":["CSV"]},{"id":"https://datahacks.dev/library/csv-analytics-export-percent-signs","url":"https://datahacks.dev/library/csv-analytics-export-percent-signs","title":"An Analytics Export With Percent Signs and Dashes Cleaned","summary":"An analytics CSV writes numbers as text: 12,430, 42.5%, $1,204.50, (4.1%) and dashes. Convert them with one macro and prove nothing is left unparsed.","content_html":"Exports from analytics tools are written for people. Visits come as 12,430, rates as 42.5%, revenue as $1,204.50, a change as +8.2% or (4.1%), and anything unavailable as a dash, two dashes, n/a or a typographic em dash, depending on which column and which day. Opened in a spreadsheet, it looks right. Loaded into a database or a script as numbers, every one of those cells fails. The cleaning is mechanical and the risk is in the details: a placeholder missed becomes a zero or an error, a parenthesis read wrongly flips a sign. This article cleans a four-row file with one function, shows the proof that nothing escaped and discusses what the placeholders mean. It is one of the problems in worked CSV problems and the patterns behind them, and a close relative of turning currency strings into numbers in a CSV.
Here is the file. Every value is display text:
\npage,visits,bounce_rate,conversion,revenue,change\n/home,\"12,430\",42.5%,3.1%,\"$1,204.50\",+8.2%\n/pricing,\"3,018\",-,--,$0.00,(4.1%)\n/blog/csv,\"8,764\",61.0%,0.4%,\"$310.00\",-1.5%\n/contact,0,n/a,n/a,—,Profile the shapes with digits replaced by 9, as in worked CSV problems, and each column states its own rules. bounce_rate has 99.9%, a lone - and a/a (the letters of n/a replaced by a). revenue has $9,999.99, $9.99, $999.99 and a single —. change has (9.9%), +9.9% and -9.9%, and one empty value. From those shapes the rules write themselves: the % and $ are decoration, the commas are thousands separators, a leading + is decoration, parentheses mean negative, and four tokens (-, --, n/a, —) plus the empty string mean no value.
Read everything as text and define two small macros: one that recognizes placeholders, one that converts a value.
\nCREATE MACRO is_placeholder(s) AS s IS NULL OR trim(s) IN ('', '-', '--', '—', 'n/a', 'N/A');\n\nCREATE MACRO num(s) AS CASE\n WHEN is_placeholder(s) THEN NULL\n WHEN s ~ '^\\(.*\\)$' THEN -1 * TRY_CAST(regexp_replace(s, '[()%$,+\\s]', '', 'g') AS DECIMAL(18,4))\n ELSE TRY_CAST(regexp_replace(s, '[%$,+\\s]', '', 'g') AS DECIMAL(18,4)) END;\n\nSELECT page, num(visits) AS visits, num(bounce_rate) / 100 AS bounce_rate, num(conversion) / 100 AS conversion,\n num(revenue) AS revenue, num(change) / 100 AS change\nFROM read_csv('analytics.csv', all_varchar=true);page visits bounce_rate conversion revenue change\n/home 12430 0.425 0.031 1204.5 0.082\n/pricing 3018 NULL NULL 0.0 -0.041\n/blog/csv 8764 0.610 0.004 310.0 -0.015\n/contact 0 NULL NULL NULL NULLWalk through the conversions. 12,430 lost its comma and became 12430. 42.5% lost the sign and, divided by 100, became 0.425, a proportion, which is the form a database should store: store 0.425 and format it as a percent when displaying. $1,204.50 became 1204.5. +8.2% became 0.082. (4.1%) matched the parenthesis rule, was stripped to 4.1, multiplied by minus one and divided by 100, giving -0.041. The placeholders became NULL, which is what \"no value\" should be.
A few details of the macros matter. The conversion uses TRY_CAST, which returns NULL for a value it cannot read instead of raising, so a new junk token does not stop the load, and for that reason the check below is not optional. The regular expression removes the characters you listed and nothing else, so a letter left in a value, such as 12.5k or 3.1x, survives and fails the cast, which is correct. The DECIMAL(18,4) type keeps four decimal places, enough for a percentage given to two. And the macro takes the whole cell as text, so the same function works on every column, which is why the cleaning is a few lines and not a script per column.
Because TRY_CAST turns anything unreadable into NULL, a NULL in the output is ambiguous: it could be a placeholder you intended or junk you did not. Separate the two by counting values that are not placeholders and still did not parse:
SELECT sum(CASE WHEN NOT is_placeholder(v) AND num(v) IS NULL THEN 1 ELSE 0 END) AS unparsed\nFROM (SELECT visits AS v FROM raw UNION ALL SELECT bounce_rate FROM raw UNION ALL SELECT conversion FROM raw\n UNION ALL SELECT revenue FROM raw UNION ALL SELECT change FROM raw);unparsed\n0Zero, on twenty cells. Wire it as a hard failure: if the count is not zero, stop and print the offending values, because it means a token you have not met, perhaps N.A., null or a currency code. A second proof is a total: the visits summed to 24,212 after cleaning, and the same figure computed by hand from the original, 12,430 plus 3,018 plus 8,764 plus 0, is 24,212. Totals that match a number published elsewhere, such as a dashboard's headline figure, are the strongest evidence that the cleaning kept the meaning.
The macros turn four different tokens into NULL, and it is worth asking whether they mean the same thing. - in a bounce rate column often means the metric could not be computed, because there were no sessions to compute a rate from. n/a may mean the same or \"not applicable for this page type\". The em dash in revenue for /contact may mean no revenue tracking is set up, while $0.00 for /pricing means tracking works and nothing was earned. Converting both to NULL loses a distinction, and converting both to zero invents data: a zero revenue and an untracked page are different facts, and an average that includes zeros for untracked pages is wrong.
So decide per column, write it down and keep the information if it matters. A cheap way is to preserve the original in a flag column: revenue_raw beside revenue, or a revenue_status with ok, no_data and not_tracked. Averages should then be over rows that have a value, with the count of rows reported. In the test file, /contact shows 0 visits and no rates: zero visits explains why a bounce rate cannot exist, and a rule of \"a missing rate with zero visits is expected\" turns a mystery NULL into a checked one.
Storing 42.5 versus 0.425 is a source of long-running confusion. Pick one. I prefer proportions (0.425) in storage, since they multiply and combine without conversion, and a column named bounce_rate holding 0.425 is clear when the type or comment says so. If the target system wants percent as 42.5, say so in the column name (bounce_rate_pct). The bug to avoid is a feed in which some rows are 0.425 and some 42.5 because the source mixed conventions: check the range, which for a proportion should lie between 0 and 1, and flag rows outside it.
If the export is the only record of why a number is missing, do not discard it. Analytics tools often use distinct tokens for distinct reasons, and a new token appearing in the file is information: a tracking change, an outage, a new page type. The unparsed check surfaces it. Add the token to the list only after you know what it means, and put the meaning in a comment. For the broader decision about missing values, see handling missing values in a CSV. The functions used are documented in DuckDB's regular expression reference and its numeric types page.
A script that edits a configuration file has an easy job and a hard constraint: leave everything else alone. The file has comments that explain why values are what they are, a key order that people chose, quotes that matter and a layout that the team agreed on. A library that loads into plain dictionaries and writes them back destroys all of it. ruamel.yaml is the Python library built for this job, and it takes a few settings to get a clean result. Here is the test with the output, and the settings.
# Service settings\nservice:\n name: \"web\" # quoted on purpose\n ports:\n - 80\n - 443\n tags: [a, b]\n note: |\n keep this\n block\n retries: 3The edit: change retries to 5, add a region key with a comment.
from ruamel.yaml import YAML\ny = YAML()\ny.preserve_quotes = True\ny.indent(mapping=2, sequence=4, offset=2)\nd = y.load(src)\nd[\"service\"][\"retries\"] = 5\nd[\"service\"][\"region\"] = \"eu\"\nd[\"service\"].yaml_add_eol_comment(\"added by script\", \"region\")\ny.dump(d, sys.stdout)# Service settings\nservice:\n name: \"web\" # quoted on purpose\n ports:\n - 80\n - 443\n tags: [a, b]\n note: |\n keep this\n block\n retries: 5\n region: eu # added by scriptThe header comment, the quoted name and its trailing comment, the flow list (with its odd spacing normalized), the block string and the order of keys are all intact. The changed value shows the new number, and the added key carries its own comment, aligned with the other trailing comment. ports is indented under its key because of indent(mapping=2, sequence=4, offset=2), which I chose to match a layout the team might use.
The loaded object behaves like a dictionary (d[\"service\"][\"retries\"]) and a list, so most code that reads or edits needs no change. The extra methods are for comments: yaml_add_eol_comment, yaml_set_start_comment and yaml_set_comment_before_after_key. Comments attach to keys and list items, and they survive when you move or modify the values around them, but not when you delete the key they are attached to.
Round-trip mode with the defaults still keeps comments and order, with small differences:
\ny2 = YAML(); d2 = y2.load(src); d2[\"service\"][\"retries\"] = 5# Service settings\nservice:\n name: web # quoted on purpose\n ports:\n - 80\n - 443\n tags: [a, b]\n note: |\n keep this\n block\n retries: 5The quotes around web are gone because preserve_quotes defaults to False, and the trailing comment moved over to column 20 because the value got shorter. The list stays at the key's indentation because that is the default, which preserves the original here. If your file has a style, set preserve_quotes = True and the indentation, and compare the result with the original through a diff before writing it. If the diff has only the intended change, the settings match your file.
yaml.safe_dump(yaml.safe_load(src), sort_keys=False)service:\n name: web\n ports:\n - 80\n - 443\n tags:\n - a\n - b\n note: 'keep this\n\n block\n\n '\n retries: 3The header comment and the trailing comment are gone, \"web\" lost its quotes, the flow list became a block list, and the block string became a single-quoted scalar with blank lines. Nothing here is a bug in PyYAML. It builds plain Python data that cannot hold comments or layout, so it writes the data in its own style. For read-only use that is fine. For editing, use ruamel.yaml or a structure-aware tool like yq, which also keeps comments (with small spacing changes) (comments in YAML and what parsers do with them).
YAML 1.2. ruamel.yaml defaults to the 1.2 rules, so NO is a string and 12:30 is not a number, and its output is written accordingly. A file that depends on 1.1 booleans will read differently. If you need 1.1 behavior, set y.version = (1, 1).
Line width. By default long lines are wrapped at 80 columns. Set y.width = 4096 to stop the library from rewrapping long values that you did not change.
Explicit document start. If the original starts with ---, set y.explicit_start = True to keep it.
Duplicate keys. Round-trip mode raises DuplicateKeyError by default, a useful side effect when editing.
Anchors and merge keys. They are preserved as anchors and aliases and reproduced on dump, and merge keys are kept as << entries. Editing a value inside an anchored mapping changes it for all aliases, as in the file's meaning.
Types. Integers keep their written form where possible (hex, octal with underscores are remembered), and floats keep their formatting, which is why a version 3.10 can be written back as 3.10 in round-trip mode, though the parsed value is still the float 3.1. Do not rely on it for versions; quote them.
Speed. On a 1.4 MB document, round-trip loading took 7.52 seconds, ruamel.yaml safe mode 5.49 and PyYAML's C loader 0.67. This is a tool for files people read, not for large data.
Load with YAML() and the settings above.
Make the minimal change in the loaded structure.
Dump to a string, not straight over the file.
Diff the string against the original. If any line other than the intended ones changed, adjust the settings or fall back to an in-place edit with yq.
Write the file, and reload it in a test to check the values.
Keep a backup or rely on version control, and commit the change with its diff.
Automating step 4 is worth the five lines. A script that edits a hundred files and refuses to write any file whose diff is larger than expected stops a formatting regression from landing in a hundred places (using yq to update many YAML files at once shows a bulk-edit workflow).
\nBeyond changing a value, scripts often need to insert a whole new block, append to a list or remove an entry. The round-trip containers support the operations you expect from dictionaries and lists, with the comment machinery carried along. Appending to a list with d[\"service\"][\"ports\"].append(8080) keeps the existing comments and adds an item in the same style. Inserting a key at a specific position uses insert(pos, key, value) on the mapping, which matters when the file has a convention for the order of keys, such as name first. Removing a key with del removes its trailing comment as well, and the comment above it stays attached to the next key, which can leave a stray comment that now describes the wrong thing. After a deletion, read the diff and move any orphaned comment.
For a new nested block, build it with the same classes the loader uses so that the dumper formats it like the rest of the file: from ruamel.yaml.comments import CommentedMap, create a CommentedMap, fill it and assign it. A plain dictionary also works, and is written in block style with the document's indentation settings. If you want a particular block to be written in flow style, call .fa.set_flow_style() on the container.
Finally, test the edit script on a copy of a real file before pointing it at a directory. Run it, diff the result and read every changed line. Round-trip libraries are good, and files in the wild contain constructs that surprise them: unusual comment placement, anchors nested in lists, tabs in comments, folded scalars with odd indentation. The diff is how you find out, and the cost is a minute.
\nThe cluster hub is reading and writing YAML in Python safely. The simpler tool for read-only loading is loading YAML in Python with PyYAML safe_load, and the writing side with PyYAML in writing YAML from Python without ugly output. Documentation: the ruamel.yaml documentation and its PyPI page. Look at the result in the YAML formatter.
\nYou need byte-identical output. No library guarantees it. Whitespace inside flow collections, comment alignment and line wrapping can change. Use text-level edits for tiny changes.
\nYou edit many files. Round-trip tools are slow per file and drift between versions. Pin the version and check diffs.
\nYou need other formats too. For TOML or JSON with comments, other tools apply.
\nThe file is generated. If a program regenerates the file on each run, comment preservation is moot. Document the generator and keep the comments there.
","date_published":"2026-10-30T16:51:00+00:00","tags":["YAML"]},{"id":"https://datahacks.dev/library/loading-multiple-documents-one-yaml","url":"https://datahacks.dev/library/loading-multiple-documents-one-yaml","title":"Loading Multiple Documents From One YAML File in Python","summary":"A YAML file can hold several documents separated by three dashes. safe_load fails on them, safe_load_all reads them and safe_dump_all writes them.","content_html":"Kubernetes manifests, multi-resource Compose overrides, CI pipeline definitions and exported datasets often put several YAML documents in one file. The separator is a line with three dashes. If you load such a file with the function you use for single documents, it fails. If you load it with the right function, you get a generator you have to treat with care. This post covers the format and the Python calls, with the output.
\nA YAML stream is a sequence of documents. A document starts at the beginning of the stream or after a line of three dashes (---), and it ends at the next ---, at a line of three dots (...) or at the end of the file:
---\na: 1\n---\nb: 2\n---\n- xThe first document is a mapping, the second another mapping, the third a list. The first --- is optional (a file without it is a single document). Each document is independent: it has its own root node, its own anchors and its own directives. A common real example is a Kubernetes file that has a Deployment document, a Service document and a ConfigMap document back to back.
docs = \"---\\na: 1\\n---\\nb: 2\\n---\\n- x\\n\"\nprint(list(yaml.safe_load_all(docs)))\ntry:\n yaml.safe_load(docs)\nexcept yaml.YAMLError as e:\n print(\"safe_load on 3 docs:\", type(e).__name__, str(e).split(\"\\n\")[0][:80])[{'a': 1}, {'b': 2}, ['x']]\nsafe_load on 3 docs: ComposerError expected a single document in the streamsafe_load_all returned the three documents: two dictionaries and a list. safe_load refused, with a ComposerError that says exactly what is wrong. Note that safe_load fails only when there is more than one document. A file with a leading --- and a single document loads fine, and a file whose second document is empty may or may not, depending on how you wrote the separator. Code that must accept either shape can use safe_load_all and take the first element, or check the length.
Another detail: a file of only comments or whitespace yields no documents from safe_load_all and None from safe_load. A file whose last line is --- yields a final document of None, which surprises code that iterates and indexes each document. Filter: docs = [d for d in yaml.safe_load_all(f) if d is not None].
safe_load_all returns a generator, and the work happens as you iterate. If the input is an open file, the file must still be open then:
with open(\"stack.yaml\", encoding=\"utf-8\") as f:\n docs = yaml.safe_load_all(f) # nothing parsed yet\n# file closed here: iterating docs now raises ValueError (I/O on closed file)\n\nwith open(\"stack.yaml\", encoding=\"utf-8\") as f:\n docs = list(yaml.safe_load_all(f)) # parsed while the file is openThe first version looks right and fails when you use docs. Either consume the generator inside the with block (for d in yaml.safe_load_all(f): ...) or turn it into a list. For very large streams the loop is better, since it holds one document at a time. A syntax error in the third document is raised while you are iterating to it, after you have already processed the first two. For all-or-nothing behavior, build the list first.
print(yaml.safe_dump_all([{\"a\": 1}, {\"b\": 2}], explicit_start=True), end=\"\")---\na: 1\n---\nb: 2safe_dump_all takes an iterable of objects and writes each as a document. With explicit_start=True every document, including the first, begins with ---, which is what most tools expect and what makes the output unambiguous. Without it, the first document has no marker. Add sort_keys=False and the other options from the single-document safe_dump (writing YAML from Python without ugly output).
An anchor defined in one document is not visible in the next:
\n---\nbase: &b {x: 1}\nuse: *b\n---\nother: *bThe second document fails with an error about an undefined alias, because b was defined in the first. If several documents share definitions, repeat them, or load the shared part separately and merge in code. The same isolation protects you: a document cannot accidentally depend on a distant definition.
yq handles streams: yq '.a' file.yaml runs the expression on every document, and yq eval-all '[.a]' -o=json -I=0 file.yaml collects across them. For a stream of two documents with a: 1 and a: 2 it printed 1, ---, 2 for the first form and [1,2] for the second. Use select(document_index == 1) to pick one. Splitting a stream into files is yq -s '.metadata.name' stack.yaml, which writes one file per document named by the expression (using yq to update many YAML files at once covers bulk workflows).
Iterate and dispatch by kind. For a Kubernetes-style stream, loop over the documents and route each by its kind key. Validate each document against its own schema.
Keep order. The order of documents in the file is meaningful for some tools (a namespace must exist before the resources in it). safe_load_all preserves it, and so should your writer.
Count and report. When a stream is malformed, report which document (by index) failed, not only the line. A counter in the loop gives you that.
\nPrefer separate files for unrelated data. If the documents have nothing to do with each other, several files are easier to diff, review and validate.
\nThree small cases behave in ways that surprise people. A file ending with a lone separator line produced an extra document: loading a: 1 followed by --- gave [{'a': 1}, None], so a loop that indexes each document fails on the trailing None. A file containing only a comment gave an empty list from safe_load_all and None from safe_load, which are different empty results. And the stream-with-leading-separator --- and one mapping loaded through plain safe_load as {'a': 1}, so a leading marker alone does not make a file multi-document.
The closed-file case is the one that bites in practice: creating the generator inside a with block and using it after the block raised ValueError: I/O operation on closed file. Because the parse is lazy, the error appears at the point of use, far from the cause. Make the habit automatic: either loop inside the block or call list(...) there.
If you validate each document, give the error the index. Wrapping the loop in enumerate(yaml.safe_load_all(f), start=1) lets you report \"document 3 of the stream is invalid\", which saves a search through a long file.
The cluster hub is reading and writing YAML in Python safely. The single-document call is explained in loading YAML in Python with PyYAML safe_load. Document markers are defined in section 9 of the YAML 1.2.2 specification, and PyYAML's functions in its documentation. Convert one document to JSON in the YAML to JSON tool.
\nThe consumer expects one document. Many tools read the first document only or reject a stream. Check before combining.
\nThe documents are unrelated. A bundle of independent things in one file makes diffs, ownership and validation harder.
\nYou need to merge them. If the intent is one configuration made of parts, use separate files and a merge step, or anchors within one document (splitting a large YAML config into several files).
\nThe data is a long list of records. A list in one document is simpler than hundreds of documents, and parsers handle it better.
","date_published":"2026-10-30T16:51:00+00:00","tags":["YAML"]},{"id":"https://datahacks.dev/library/csv-import-export-features-applications","url":"https://datahacks.dev/library/csv-import-export-features-applications","title":"CSV Import and Export Features in Applications Done Right","summary":"Build CSV import and export into an application: staged imports with dry runs, streamed and background exports, column names, round trips and safety.","content_html":"Almost every business application gets two requests within its first year: \"can I upload a spreadsheet to create these in bulk\" and \"can I download this list as a CSV\". They sound small. The first turns out to be a data-quality problem with a user interface, and the second a performance problem that appears the day a customer has enough rows. Both are shaped by the same fact: a CSV is a file written by humans and other programs, in whatever dialect they like, and an application that treats it as a clean table will find out otherwise from the first user. This guide describes the import and export features I would build, stage by stage, with real runs of the parts that can be tested without a framework. It covers the design of the import screen, previews and dry runs, error reports, streaming and background exports, column names, round trips and the safety checks, and it points to the focused article for each. The back-end side of repeated imports from a partner is a different problem, covered in automating CSV imports into a database.
\nThe version of import that most teams build first is a form with a file field and a submit button. It reads the file, inserts rows and shows \"done\", and it fails in a way users cannot understand the first time a cell contains a surprise. The version that works has stages, and each stage gives the user a chance to stop or fix something before the next:
\n| Stage | What happens | What the user sees |
|---|---|---|
| Upload | File arrives, size and type checked, stored outside the web root | A progress bar, a clear refusal for a file that is too large |
| Inspect | Encoding, delimiter, header and row count detected | \"We found 4 columns and 2,310 rows\" |
| Map | File headers are matched to the fields the application knows | A table of file columns and target fields, editable |
| Validate | Every row is checked, nothing is saved | A summary: ready, warnings, errors, with examples |
| Commit | Valid rows are written in a transaction | A confirmation and a count |
| Report | What was created, updated and rejected, with reasons | A downloadable file of the rows that need fixing |
Every article in this cluster is one stage. Designing a CSV import screen users can succeed with covers the screens, previewing a CSV upload before committing the dry run, and reporting row-level errors back to the user the report. The point of the table is the order. Validation comes before commit, and nothing is saved until the user has seen what will happen.
\nHere is the core of such a pipeline in plain PHP with SQLite, small enough to read in a minute. The input is a file of six customers, two of whom already exist, with one row that has an unknown plan and no seat count, one with a bad email and one with a plan the application does not offer:
\nEmail,Full name,Plan,Seats\nada@example.test,Ada Lovelace,pro,5\nbob@example.test,Bob Stone,free,1\ncy@example.test,Cy Young,enterprise,\nnot-an-email,Di Park,pro,3\ned@example.test,Ed Fox,gold,2\nfay@example.test,Fay Moss,team,12The import function validates each row, inserts or updates inside a transaction and commits or rolls back according to a flag. The validation is deliberately plain: an email must look like an email, a plan must be one of free, pro and team, seats must be a whole number of at least one.
function runImport(PDO $pdo, string $path, bool $commit): array {\n $pdo->beginTransaction();\n $summary = ['created' => 0, 'updated' => 0, 'rejected' => 0];\n $report = [];\n $in = fopen($path, 'r');\n fgetcsv($in, 0, ',', '\"', ''); // header\n $line = 1;\n while (($f = fgetcsv($in, 0, ',', '\"', '')) !== false) {\n $line++;\n $row = ['email' => trim($f[0]), 'name' => trim($f[1]), 'plan' => trim($f[2]), 'seats' => trim($f[3])];\n $errors = validate($row);\n if ($errors) {\n $summary['rejected']++;\n foreach ($errors as [$col, $val, $msg]) $report[] = [$line, $col, $val, $msg];\n continue;\n }\n // update when the email exists, insert otherwise ...\n }\n $commit ? $pdo->commit() : $pdo->rollBack();\n return [$summary, $report];\n}Run first as a dry run against a table that already holds two of the customers, with the commit flag off:
\nbefore: 2 customers\ndry run: {\"created\":1,\"updated\":2,\"rejected\":3} -> table still has 2\nerror report:\nline,column,value,problem\n4,plan,enterprise,\"plan must be one of: free, pro, team\"\n4,seats,,\"seats must be a whole number of 1 or more\"\n5,email,not-an-email,\"not a valid email address\"\n6,plan,gold,\"plan must be one of: free, pro, team\"One customer would be created, two updated and three rejected, and the table still has its two original rows, because the transaction was rolled back. Then the same call with commit on:
\ncommit: {\"created\":1,\"updated\":2,\"rejected\":3} -> table now has 3\n ada@example.test | Ada Lovelace | pro | 5\n bob@example.test | Bob Stone | free | 1\n fay@example.test | Fay Moss | team | 12The dry run and the commit are the same code path, and the only difference is the last line. That is the design to copy: a preview that runs different code from the real import will drift, and the day it says \"all clear\" and the real import fails is the day nobody trusts either. Note what the report says. Line 4 has two problems, and both are listed, since a user who is told one error per row will fix, re-upload and meet the second. Lines are physical line numbers in the user's file, with the header as line 1, so they can find the row in a spreadsheet.
\nAn error message is a product feature. \"Import failed\" is useless, and \"Row 4: plan 'enterprise' is not one of free, pro, team\" can be fixed in ten seconds. Four rules make the report usable. State the row and the column, in the user's terms: the file's own header (Plan) and not your internal field name. Show the value that failed, so a stray space or a wrong character is visible. Say what would be accepted. And give the whole list in a form the user can work with, which means a CSV of the rejected rows with an added error column, so the user fixes those rows in place and uploads only them (reporting row-level errors back to the user has the format). Cap what you display on screen at the first hundred errors and group them by type, with the full list in the download, because a file with 40,000 bad rows is not one that a person fixes row by row, and the pattern is more useful than the list.
The first export feature is usually written as: query the rows, build the CSV in a string or an array, return it. It works in testing with 200 rows. At a million rows, it fails. The experiment: produce 1,000,000 rows of four columns (a 40.7 MB file) two ways in PHP, one collecting every row into an array and then writing, one writing each row as it is produced.
\narray rows=1000000 time=1.43s peak memory=336.0 MB file=40.7 MB\nstream rows=1000000 time=1.09s peak memory=2.0 MB file=40.7 MBThe array version peaked at 336 MB, and under PHP's default memory_limit of 128 MB it did not finish at all: Fatal error: Allowed memory size of 134217728 bytes exhausted. The streaming version used 2 MB, whatever the row count, and was faster. The same file, two orders of magnitude apart in memory. The principle is general: produce the file row by row, write each row to the response or a file as soon as you have it, and never hold the whole result. In a web framework that means a streamed response with a generator (Laravel's streamDownload, Django's StreamingHttpResponse, Rails' ActionController::Live or a plain enumerator body). Streaming a CSV export from a web app goes through the headers and the buffering traps.
Streaming solves memory. It does not solve time: a request that runs for ninety seconds may hit a proxy timeout, hold a database connection and tie up a worker. For large exports, hand the work to a background job that writes the file and sends the user a link, as in background jobs for large CSV exports. The rule I use: a synchronous download for results the user can see in a few seconds, and a job beyond that, with the threshold set from measurement.
\nWhether streamed or in a job, a large export reads the table in pieces. The obvious way is LIMIT 10000 OFFSET n, advancing n by 10,000. It gets slower as n grows, because the database has to walk past n rows to find where to start. The alternative is keyset pagination: remember the last key you saw and ask for WHERE id > last ORDER BY id LIMIT 10000. On a million-row SQLite table, in chunks of 10,000:
offset: total 0.75s, first chunk 3.9 ms, last chunk 11.8 ms\nkeyset: total 0.28s, first chunk 3.2 ms, last chunk 2.7 msThe offset version took 2.7 times as long overall, and its last chunk cost three times its first. SQLite is a gentle case, since it walks an index cheaply, and in MySQL or PostgreSQL with a sort that is not covered by an index, the penalty is larger. Keyset stays flat. It needs a unique, ordered key, and a result sorted by something else (a name, a date) needs a composite key, such as (date, id). It also has a consistency benefit: rows added during a long export do not shift the pages, so nothing is read twice or skipped.
An export and an import are two halves of a contract, and the contract is in the details. The header row is the interface. Use names a user can recognize, not column names from your schema, and accept variants on import: E-mail, Email address and email_address should all map to the same field. A header-mapping function in PHP, run on three real-looking headers, did it:
[\"E-mail\",\"Full Name\",\"Tier\",\"Number of Seats\"] -> mapped {\"email\":0,\"name\":1,\"plan\":2,\"seats\":3}; missing []; unknown []\n[\"email_address\",\"contact\",\"plan\",\"seats\",\"notes\"] -> mapped {\"email\":0,\"name\":1,\"plan\":2,\"seats\":3}; missing []; unknown [notes]\n[\"Email\",\"Name\",\"Plan\"] -> mapped {\"email\":0,\"name\":1,\"plan\":2}; missing [seats]; unknown []The first header has a byte order mark and a double space, and still mapped. The second has an extra column the importer does not know and reported it as unknown instead of failing. The third is missing a required column and said which. Choosing column names users recognize covers the naming rules.
\nTypes need decisions that a spreadsheet user cannot see. Dates in ISO 8601, so that a file opened in a European Excel and saved again does not flip day and month (handling time zones in CSV timestamps). Booleans as true and false, or 1 and 0, one convention, documented. Money as a decimal with a point and no symbol. Identifiers as text, so that a zip code keeps its zero. And nulls: decide what an empty cell means, and what an empty string means, and whether the two differ.
A well-designed export can be fed back into the import and produce the same data, so that a user can export, edit in a spreadsheet and re-import. Test that property, and test it on the data. I exported a table in which one note was NULL and another was an empty string, imported the file into a second table and exported again. The two exports were byte-for-byte identical:
\nsecond export identical to first: true\nrows that differ after the round trip: 1\n id 1 note before NULL after ''\n id 2 note before '' after ''The files matched and the data did not. The CSV has no way to tell a NULL from an empty string, since fputcsv writes both as nothing, so the import chose empty string and the NULL was gone. Setting the import to read an empty cell as NULL fixed row 1 and broke row 2, where an empty string became NULL. Neither is wrong, and a decision is required: say what an empty cell means in the format's documentation, and if the distinction matters, represent it in the file, with a marker for NULL or by quoting the empty string. A round-trip test that compares files will pass while the data drifts, so compare the tables. Round tripping exports back into imports goes through the other pitfalls.
Users double-click, refresh the page, upload the same file again to be safe and send the corrected file on top of the wrong one. An import feature that has no answer to \"this file again\" creates duplicates, and the cleanup is a support ticket. Three layers of protection work together. The first is a record of each import: a row in an imports table with the user, the file name, a content hash, the status (previewed, committed, failed), the counts and the times. A commit button that reads \"commit import 41\" instead of \"commit this file\" cannot be pressed twice, because the second press finds the import already committed. The second is a business key on the rows, so a repeat updates what is there and does not insert again, which is what the updated count in the dry run shows: two of the six rows matched existing customers by email. The third is the content hash, which lets you say \"you uploaded this exact file ten minutes ago\" before doing any work, with a button to proceed anyway for the case where it was deliberate (idempotent CSV imports that are safe to run twice goes through the designs).
A related decision is what an import does to rows that are in the application but not in the file. Most features should leave them alone: the file is a set of changes, not a replacement of the whole table. A \"replace everything\" mode is useful and dangerous, and should be a separate, named option with a confirmation that states how many rows will be removed.
\nThe transaction in the demo wraps six rows. For six thousand, it still works, and for six million it holds locks and a long-lived transaction that other users will feel. The common compromise is to validate the whole file first, without writing, and then commit in batches of a few thousand rows, each its own transaction, in a background job. Batches change the failure semantics: if the job dies at row 3 million, the first 2.9 million are committed. Decide whether that is acceptable and say so on the screen. For an all-or-nothing import, load into a staging table first, validate there, and move the rows into the real tables in one statement (loading a CSV into a staging table, then merging). For an import where partial progress is fine, record which batches completed so that a rerun resumes.
\nProgress deserves honest treatment. A file of two hundred thousand rows takes minutes, and a spinner with no end is worse than a bar. Count the rows in the inspect stage, update a progress record after each batch, and let the page poll it. Let the user leave the page and come back, with the import visible in a list. And let them cancel: the job checks a flag between batches. Both are small additions and they stop the import screen from becoming the place where people wait and wonder.
\nUsers edit these files in the locale of their machine, and the file records it. A German Excel writes 1.234,50 and a semicolon, a French one writes dates as 04/03/2026 meaning the fourth of March, a Japanese one may save in Shift JIS if the user chose the wrong format. Accept what you can: detect the delimiter from the first lines (telling what delimiter an unknown CSV file uses), accept UTF-8 with or without a byte order mark and offer an encoding choice for the rest. For numbers and dates, the safest rule is to tell the user the format you expect and show it in the template, with ISO dates and a decimal point, then reject ambiguous values with a message that names the format, since guessing a date order is how a quiet error becomes a wrong invoice (a bank CSV with dates in two formats shows what evidence you can use).
If your interface is translated, the headers of the export should be too, and the importer should accept both the translated and the English names, since a file exported in one language and re-imported by a colleague who uses another is an ordinary event. Keep an internal key for each column, translate only the label, and map on either.
\nThe most valuable feature after the error report is \"undo this import\". It is cheap if you plan for it: store the import ID on every row the import created or changed, and keep the previous values of the rows it updated. Undo then deletes the created rows and restores the old values of the updated ones, and a user who realizes that the file was the wrong one has a way back that does not involve a support request and a database restore. Even without undo, the import record, with who, when, the file, the counts and the error report, is the audit trail that answers \"who changed these customers\".
\nBe careful with what you keep. The uploaded file may hold personal data, and the import record holds a copy of it. Keep it as long as undo or audit needs it and delete it afterward, on a schedule you can state (handling CSV files full of customer data under GDPR).
\nAn import writes data, and an export reads it, and both must obey the same rules as the screens. Check three things. Who may import: bulk creation is a power, and many applications reserve it for administrators or a specific role, with the permission checked on the server, never only by hiding the button. Which rows the import may touch: an import that updates by email must be scoped to the importing user's account or tenant, so that a file containing another tenant's email cannot overwrite their record, and the lookup in the update step must include the tenant condition. And which columns: a user who cannot edit a price in the interface must not be able to change it through a column in a file, so the mapping stage should offer only the fields the user may write.
\nThe export has the mirror-image risk. The CSV of a list view must apply the same filters, the same row-level permissions and the same column masking as the list. An export that is built from a separate query by a separate developer is the classic route around access control: the screen hides the salary column and the CSV includes it. Build both from one definition of what the user may see, and test the export as a restricted user. Rate-limit exports as well, since a complete data dump is a one-click operation that deserves a log entry and a limit.
\nThe cheapest way to reduce errors is to give users a file to start from. A template download with the exact header row, an example row and, if you can, a second sheet of rules (allowed values, formats) prevents mismatched headers, wrong date formats and the use of the wrong column order. Generate it from the same column definitions that drive validation, so they cannot disagree, and version it: when you add a required column, the template changes, and an old file with the old header should get a message that says which template it came from and what is missing.
\nOffer the export as the template, too. A user who wants to bulk-update existing records starts from an export of those records, edits the cells and uploads the file. That workflow makes the round trip a first-class feature, which is why the round-trip tests matter: if exporting and re-importing without edits changes anything, the feature teaches users not to trust it.
\nOnce it is live, watch it. The numbers worth tracking are small. The share of imports that commit on the first attempt, which says how well the screen and the template guide people. The most common error codes, which tell you where the instructions are unclear or where a validation rule is stricter than users expect. The size of imports and exports, and their duration, which tell you when to move to batches or a job. And the number of support requests that mention a file, which is the metric the rest exist to reduce. A feature that fails for 30 percent of first attempts with the same error is a feature with a defect in its explanation, and the cheapest fix is a better message or a template, not more validation.
\nImport and export are attack surfaces, and the checks are the ones in CSV security risks that catch teams out. On export, any text a user typed that starts with =, +, - or @ must be neutralized for spreadsheets (escaping formula characters in a CSV export), because the person who opens the file is not the person who typed the value. On import, enforce limits on size, rows, columns and field length before you parse (limiting file size and row count for CSV uploads), and process the file under a role that can insert only what the user may create. Permissions and tenancy belong in the commit stage: an import into one customer's account must never touch another's rows, and an export must apply the same row-level filters as the screen it came from, or it becomes the way around them. Record who imported what and when, since a bulk import is the easiest way to change a thousand records at once and the first thing someone asks about afterward.
Most people who touch these files use Excel, and a feature that ignores that gets support tickets. Offer an Excel-friendly export, UTF-8 with a byte order mark so accents survive a double-click, and say so on the button (Excel CSV UTF-8 versus CSV comma delimited). Accept both comma and semicolon delimiters on import, since a European Excel writes semicolons (opening a semicolon separated CSV in Excel). Offer a template download with the exact headers and an example row, which prevents more errors than any validation. And assume that identifiers will be mangled if a person edits the file in a spreadsheet: leading zeros and long numbers are the usual casualties (opening CSV files in Excel and Sheets without damage), so validate them on import and say what went wrong.
\nThe tests that earn their keep are few. A fixture of awkward files: with a byte order mark, with semicolons, with quoted newlines, with every error type, empty, header only, and one that is too large. A dry-run test that asserts the table is unchanged afterward. A commit test that asserts the counts. A round-trip test that compares tables. An error-report test that asserts the line numbers refer to the physical lines of the uploaded file. And a large-file test that exports a million rows and asserts that memory stays below a figure, since a streaming export that someone later turns into an array will pass every small test and fail in production.
\nCSV import suits bulk loading of records people already have in a spreadsheet and occasional bulk edits. It is the wrong interface when the data comes from another system on a schedule: an API or a managed connector gives you typed fields, authentication, retries and no dialect problems. It is the wrong interface for editing a few records, where a form is better, and for data with relationships, orders with line items, where a flat file cannot express the structure without conventions that are fragile. And it is the wrong export when the consumer is a program: JSON or an API preserves types and nesting, and a CSV makes the consumer guess.
\nWhere it is right, build it as described: staged, previewed, streamed, tested on round trips and defended. The framework-specific tools, queues, streamed responses and validators, differ, and the shape does not. The format is defined in RFC 4180, the PHP functions in the fgetcsv documentation, and the upload controls in OWASP's file upload cheat sheet.
A bank statement arrives as a CSV with a Date column, and half the values look like 2026-03-04 while the other half look like 05/03/2026. Someone's tool wrote both, maybe because the file was assembled from two exports, or a person typed some rows into a sheet, or the bank changed formats mid-month. Reading it with one format fails on the other half, and letting a library guess will pick a style per file or per row and not tell you which. The way through is to treat the column as two sub-problems: recognize the styles, then find proof for the order of the ambiguous one. This is the first of the problems in worked CSV problems and the patterns behind them, and the general case of dates is in fixing mixed date formats in a CSV.
Six transactions. Three dates are ISO, three are slashes:
\nDate,Description,Amount\n2026-03-04,CARD PAYMENT ACME,-120.50\n05/03/2026,TRANSFER BETA,-89.90\n2026-03-06,SALARY,2500.00\n13/03/2026,CARD PAYMENT CY,-15.00\n2026-03-14,REFUND,45.00\n03/04/2026,CARD PAYMENT DELTA,-200.00Read it as all text first. The profile of the shapes, with each digit replaced by a 9, comes from one query:
\nSELECT regexp_replace(\"Date\", '[0-9]', '9', 'g') AS shape, count(*) AS n, min(\"Date\") AS example\nFROM read_csv('bank_dates.csv', all_varchar=true) GROUP BY 1 ORDER BY n DESC;shape n example\n9999-99-99 3 2026-03-04\n99/99/9999 3 03/04/2026The ISO shape is unambiguous: year first, then month, then day. The slash shape is the question. 05/03/2026 is the 5th of March in a day-first style and the 3rd of May in a month-first one. A parser with a locale default will choose one for every slash date, and if it chooses wrongly, the error is silent: both readings are valid dates.
The statement itself holds the answer. A month cannot be above 12, so any slash date whose first part is 13 or more can only be day-first, and any whose second part is above 12 can only be month-first. Parse every slash date both ways and count the failures:
\nSELECT n, \"Date\",\n try_strptime(\"Date\", '%d/%m/%Y')::DATE AS as_dmy,\n try_strptime(\"Date\", '%m/%d/%Y')::DATE AS as_mdy\nFROM raw WHERE \"Date\" ~ '^\\d{2}/\\d{2}/\\d{4}$';n Date as_dmy as_mdy\n2 05/03/2026 2026-03-05 2026-05-03\n4 13/03/2026 2026-03-13 NULL\n6 03/04/2026 2026-04-03 2026-03-04Row 4 parses only as day-first: month-first gives NULL, since there is no month 13. That one row is the proof, and a summary query says it cleanly: the number of slash dates that fail %d/%m/%Y was 0, and the number that fail %m/%d/%Y was 1. If the styles are consistent within the file, and files are consistent far more often than not, every slash date is day-first. Row 2 is the 5th of March and row 6 is the 3rd of April.
That inference rests on an assumption, that the writer used one convention for all slash dates, and a second, independent check is cheap: the order of the statement.
\nWith the order established, the conversion tries each format in turn. try_strptime returns NULL on a mismatch instead of raising an error, so coalesce picks whichever format works:
SELECT n, coalesce(try_strptime(\"Date\", '%Y-%m-%d'), try_strptime(\"Date\", '%d/%m/%Y'))::DATE AS date,\n \"Description\" AS description, \"Amount\"::DECIMAL(10,2) AS amount\nFROM raw ORDER BY n;n date description amount\n1 2026-03-04 CARD PAYMENT ACME -120.50\n2 2026-03-05 TRANSFER BETA -89.90\n3 2026-03-06 SALARY 2500.00\n4 2026-03-13 CARD PAYMENT CY -15.00\n5 2026-03-14 REFUND 45.00\n6 2026-04-03 CARD PAYMENT DELTA -200.00Read the dates down the column: 4, 5, 6, 13, 14 March, then 3 April. A bank statement is in date order, and these are. That is a second proof, and a strong one for row 6: read as month-first, 03/04/2026 would be 4 March and sit out of order between 14 March and the end of the file. A check in SQL asserts it, comparing each date with the one before:
SELECT count(*) FROM (SELECT date < lag(date) OVER (ORDER BY n) AS back FROM clean) WHERE back;It returned 0. Put that check into the script as a hard failure, so that a new statement where it fails is stopped and not silently loaded. Also check that no date is NULL after the conversion, which would mean a third style the script does not know, such as 4 Mar 2026 or 04.03.26.
Row 2 deserves a second thought, since 05/03/2026 is valid both ways and nothing in the row decides. It is decided by the rest of the file. Under the day-first rule it is 5 March, between 4 and 6 March, in order. Under month-first it would be 3 May, and the statement would jump forward two months and back. The ordering check settles it. This is the general technique for ambiguous dates: use any constraint the data offers, such as monotonic order, a known range of dates (a statement for March should not hold dates in May) or another column that carries the date unambiguously, such as a posting timestamp. Constraints that agree are evidence, and a reading that violates one is probably wrong.
Be aware that an ordering test works only when the file is ordered. A file sorted by amount or by description gives no information, and you must not assume order without checking it on the unambiguous rows first: sort the ISO rows by position and confirm they are non-decreasing, and if they are, the assumption holds for the file.
\nSometimes there is no proving row. A statement for the first week of a month has slash dates like 03/04/2026 and 05/04/2026, all with a first and second part below 13, and the order cannot be inferred from the values. If the file is also not in date order, there is nothing left. The honest options are to ask the source (\"is this day-first?\"), to take the convention from the system that produced it (a bank in a day-first country writes day-first), or to reject the file with a message that names the ambiguity. Do not choose by the locale of the machine that happens to run the script. That is how a statement is read correctly on one laptop and wrongly in a pipeline on another.
Also decide what to do when a file mixes conventions within the slash style, for instance when two exports were concatenated, one day-first and one month-first. The proof then fails in the other direction: some rows can only be day-first and others only month-first. A file that contains both 13/03/2026 and 03/14/2026 is inconsistent, and the right response is to stop, since no single rule reads it. If you can tell which rows came from which export, for example by a column or by their position, convert each block with its own format.
The cleanest fix is upstream: ask the bank or the tool for ISO 8601 dates, which have one reading everywhere. In the meantime, the profile query, the two-way parse and the order check turn a guess into a derivation. The functions are documented in DuckDB's date format reference, and the surrounding approach in handling time zones in CSV timestamps and reconciling a bank statement CSV with your ledger, which uses the clean statement as input. The ISO format itself is defined in ISO 8601 as summarized by the W3C.
","date_published":"2026-10-30T16:51:00+00:00","tags":["CSV"]},{"id":"https://datahacks.dev/library/csv-ecommerce-export-commas-inside-product","url":"https://datahacks.dev/library/csv-ecommerce-export-commas-inside-product","title":"An Ecommerce Export With Commas Inside Product Names","summary":"A product export writes names with commas and no quotes, so rows have 6 fields, not 4. Rejoin the middle fields from the good ends and verify every row.","content_html":"A shop exports its catalog to CSV, and the product names contain commas: Cable, USB-C, 2 m, Stand, aluminium, adjustable. The export did not quote them. A line meant to have four fields now has six, every column after the name is shifted, and the loader either rejects the line or loads the price into the stock column. This is the unquoted-delimiter problem, one of the commonest damage patterns, and it is fixable when the damaged column is the only free-text one. This article diagnoses it, repairs it with a short script and shows where the repair stops being safe. It is one of the problems in worked CSV problems and the patterns behind them, and the quoting rules it breaks are explained in how CSV quoting works.
Here is the file. Look at lines 2 and 4:
\nsku,name,price,stock\nA100,Cable, USB-C, 2 m,9.90,120\nA101,Mouse,14.50,35\nA102,Stand, aluminium, adjustable,39.00,8\nA103,\"Hub, 4 port\",24.00,60The header has four fields. The first thing to check is how many each line has, through a CSV parser, not by counting commas yourself:
\nimport csv\nprint([len(r) for r in csv.reader(open(\"products.csv\"))])[4, 6, 4, 6, 4]The header, A101 and the quoted A103 have four fields. A100 has six, and so does A102: two and two extra commas. The extra fields equal the extra commas in the name, which is what unquoted commas look like. Line 5 shows the file is not uniformly broken: the exporter quoted \"Hub, 4 port\" correctly, perhaps because that name came through a different code path or a newer version. A file that mixes quoted and unquoted commas is common, and it means a fix that assumes every comma is unquoted would be wrong, so the script has to handle the already-quoted rows without damaging them.
The structure is known even though the commas are ambiguous. The first field is a SKU, a short code with no commas. The last two fields are a price and a stock count, numbers with no commas. Everything between them is the name, however many pieces it was split into. That rule is stated in one sentence, and it is the specification of the repair: take row[0] as the SKU, row[-2] and row[-1] as price and stock, and join row[1:-2] with commas as the name.
import csv, sys\n\nEXPECTED = 4 # sku, name, price, stock\nwith open(sys.argv[1], newline=\"\", encoding=\"utf-8\") as f, open(sys.argv[2], \"w\", newline=\"\", encoding=\"utf-8\") as out:\n reader = csv.reader(f)\n writer = csv.writer(out)\n header = next(reader)\n writer.writerow(header)\n for line_no, row in enumerate(reader, start=2):\n extra = len(row) - EXPECTED\n if extra < 0:\n sys.exit(f\"line {line_no}: only {len(row)} fields\")\n # The first field and the last two are well behaved; everything between belongs to the name.\n sku, middle, price, stock = row[0], row[1:len(row) - 2], row[-2], row[-1]\n name = \",\".join(part for part in middle).replace(\" ,\", \",\")\n writer.writerow([sku, name, price, stock])\n if extra:\n print(f\"line {line_no}: rejoined {extra + 1} pieces into {name!r}\")line 2: rejoined 3 pieces into 'Cable, USB-C, 2 m'\nline 4: rejoined 3 pieces into 'Stand, aluminium, adjustable'The script refuses a line with fewer than the expected fields, which would be a different kind of damage, and prints every repaired line with the name it produced, so a person can read the result. It writes through csv.writer, which adds the quotes the exporter should have added. The output file:
sku,name,price,stock\nA100,\"Cable, USB-C, 2 m\",9.90,120\nA101,Mouse,14.50,35\nA102,\"Stand, aluminium, adjustable\",39.00,8\nA103,\"Hub, 4 port\",24.00,60Every row now has four fields. The already-quoted A103 went through unchanged: the parser had read it as four fields, so extra was zero and the name was Hub, 4 port as before. The rows that were wrong now have the names they should.
Run the field-count check on the output: all lines should be 4. Then check the values against what you know about each column. The prices should all parse as decimals and the stock as integers, which is a stronger test than the field count, because an off-by-one repair would put a name fragment where a number belongs and the cast would fail. In SQL, TRY_CAST(price AS DECIMAL(10,2)) IS NULL should return no rows. Compare the row count with the input's, and if the shop can tell you the number of products, compare with that. And read a sample of the names: the repaired ones are the ones to eyeball, and the printed lines are the list.
One detail in the script deserves comment. The line .replace(\" ,\", \",\") normalizes a space before a comma. It was added for names where an exporter had padded; leave out any such cosmetic step unless you know the data needs it, since every extra rule is something you have to justify. The version in the repository should show only the rejoin and say what it assumes.
The anchor rule works only if the commas can appear in exactly one column and the columns around it have a recognizable shape. Three situations break it.
\nTwo free-text columns: if the file also had a description column with commas, you could not tell where the name ends and the description begins. A100,Cable, USB-C,Short cable, cheap,9.90,120 could split four ways, and no program can choose. The only fixes are a convention you can verify (names never contain more than two commas) or a re-export with quotes.
A free-text column at an end: if the name were the last column, there would be no right-hand anchor, and the number of pieces to join would be unknown. A left anchor of known width only works if there is a fixed count of columns after the name.
\nA name that ends in something that looks like the anchor shape: Cable, 9.90 m has a piece that parses as a number, and the right-anchored rule still works because it counts from the right by position and does not look at what the pieces contain. A rule that scans from the right until it finds \"something numeric\" would break here, and positions are safer than patterns.
For files where the repair is ambiguous, quarantine the rows: write the lines that cannot be repaired confidently to a rejects file with their line numbers and let a person decide (logging rejected rows from a CSV import for review).
\nThe repair is a patch for an exporter that does not follow RFC 4180, which asks for fields containing commas to be quoted. Tell whoever owns the export, with the line numbers and the field counts, and ask for quoting. Most platforms have a setting or a newer export format that does it correctly, and sometimes the problem is that someone saved the export from a spreadsheet that dropped the quotes. Until then, put the field-count check at the door of your pipeline so the next change, a different delimiter in a product name, a second free-text column, is caught as an error and not loaded as shifted data (checking row and column counts in a CSV). The Python module used here is described in the csv reference.
","date_published":"2026-10-30T16:51:00+00:00","tags":["CSV"]},{"id":"https://datahacks.dev/library/csv-crm-export-multi-line-notes","url":"https://datahacks.dev/library/csv-crm-export-multi-line-notes","title":"A CRM Export With Multi Line Notes and How to Parse It","summary":"A CRM export has notes with line breaks inside quoted fields, so line counts and grep mislead. Parse by records, then flatten the breaks or keep them.","content_html":"CRM exports are where multi-line fields live. A salesperson types a note with a few lines, the system exports it faithfully inside quotes, and every tool downstream that thinks in lines now disagrees about how long the file is. wc -l says 7, the importer says 3 customers, a script that processes line by line sees notes broken in half and grep returns half a record. Nothing is wrong with the file: quoted line breaks are valid CSV. What is wrong is the assumption that a line is a record. This article shows how to count and parse such a file correctly and how to flatten it when a tool downstream insists on one line per row. It is one of the problems in worked CSV problems and the patterns behind them, and it relies on the quoting rules in how CSV quoting works.
Three customers. The first note runs to three lines, the third has a quoted word inside it:
\nid,company,owner,note\n1,Acme Ltd,ada,\"Called on Monday.\nWants a quote for 40 seats.\nFollow up Friday.\"\n2,Beta Corp,bob,No answer\n3,\"Cy Foods, Inc\",cy,\"Said \"\"maybe\"\" twice\nasked for pricing\"Count it two ways:
\nwc -l crm.csv 7 crm.csvimport csv\nrows = list(csv.reader(open(\"crm.csv\", newline=\"\")))\nprint(\"records:\", len(rows) - 1)records: 3Seven lines, three records, one header. The lines break down as the header (1 line), customer 1 with its three-line note (3 lines), customer 2 (1 line) and customer 3 with a two-line note (2 lines). Any process that counts, samples or splits by line will be wrong by exactly the number of embedded breaks. That includes head -n 100, which can cut a record in half at the end of its output, and split -l, which can cut one in half between two files (splitting a large CSV into smaller files shows how to cut on record boundaries).
The CSV parser follows the quotes, so a line break inside quotes is data. In Python, open the file with newline=\"\" so the module receives the line endings unchanged and can tell a break inside a field from one that ends a record:
import csv\nwith open(\"crm.csv\", newline=\"\", encoding=\"utf-8\") as f:\n for row in csv.DictReader(f):\n print(row[\"id\"], repr(row[\"note\"]))1 'Called on Monday.\\nWants a quote for 40 seats.\\nFollow up Friday.'\n2 'No answer'\n3 'Said \"maybe\" twice\\nasked for pricing'Each note arrived as one string, with \\n where the breaks were. The doubled quotes in note 3 became single quotes: \"\"maybe\"\" inside a quoted field is the escape for a literal quote, so the value is Said \"maybe\" twice. Leaving newline=\"\" off is the classic mistake: Python's text mode translates line endings on read, and on a file with Windows line endings you may get doubled or lost breaks inside fields.
DuckDB does the same without any setting, and its read_csv detects the quoting:
SELECT id, company, replace(note, chr(10), ' | ') AS note,\n length(note) - length(replace(note, chr(10), '')) AS breaks\nFROM read_csv('crm.csv') ORDER BY id;id company note breaks\n1 Acme Ltd Called on Monday. | Wants a quote for 40 seats. | Follow up Friday. 2\n2 Beta Corp No answer 0\n3 Cy Foods, Inc Said \"maybe\" twice | asked for pricing 1Three rows, with the breaks counted per note (2, 0 and 1) and shown as | for display. The count of breaks per row also reconciles the numbers from before: 3 breaks in total, so the physical lines are 4 (the header plus three single-line records) plus 3 breaks, which is 7.
Some destinations cannot take line breaks inside a value: a log shipper, a tool that reads the CSV with split('\\n'), a fixed-format feed. Do not try to fix those by regex on the raw file, since a regex cannot know whether a line break is inside quotes. Parse first, then replace in the parsed value:
import csv, sys\nwith open(\"crm.csv\", newline=\"\", encoding=\"utf-8\") as f, open(\"crm-flat.csv\", \"w\", newline=\"\", encoding=\"utf-8\") as out:\n reader, writer = csv.reader(f), csv.writer(out)\n for row in reader:\n writer.writerow([c.replace(\"\\r\\n\", \" | \").replace(\"\\n\", \" | \").replace(\"\\r\", \" | \") for c in row])The separator | keeps the structure visible, and a space would glue sentences together (Monday.Wants). Replace \\r\\n first, then the lone \\n and \\r, so that a Windows break is not turned into two separators. If the exact text matters later, keep the original column and add the flattened copy next to it, rather than replacing it. And tell the people who read the flat version that the pipes are line breaks, because | may also occur in the original text; pick a separator that does not.
In the other direction, a tool that must receive the breaks, such as a text area, can have them back by reversing the replacement, which only works if the separator is unambiguous.
\nSpreadsheets handle the breaks, with caveats. Excel shows a multi-line cell as one cell with wrapped text if the wrap setting is on, and on a double-click open it may display only the first line. Sorting or filtering is unaffected, and saving writes them back inside quotes (Excel CSV versus RFC CSV differences). grep and awk see lines, so a search for \"Friday\" prints the third line of note 1 and not the customer it belongs to; use csvgrep or qsv search, which parse the CSV and print the whole record (working with CSV files in the shell). Database loaders handle quoted breaks if told the format is CSV: PostgreSQL's COPY ... CSV does, and its text format escapes the break as \\n instead (PostgreSQL CSV versus text format in COPY).
Parallel readers deserve a warning. A loader that splits a large file into byte ranges and parses each chunk independently can start a chunk in the middle of a quoted field and read the rest as a different structure. Systems that do this either need allow_quoted_newlines style settings and give up parallelism or must verify the split, so check the documentation before loading a 20 GB file of notes with a parallel loader.
All of this assumes the file's quoting is correct, and exports from some CRM systems are not. Check three things. Every record should have the same number of fields when parsed, which the field-count test finds: a note with a stray unmatched quote swallows the following records, and the count of records comes out low. Look at the record count against what the CRM says it exported. And scan the notes for signs of a broken field: a note that starts with a customer ID and a comma, where the parser merged two records. If the quotes are broken, the problem is the unescaped-quote case, and the repair is a lenient parser setting, shown in a log export with unescaped quotes, applied with a count to prove it.
\nFinally, consider whether the notes should be in the file at all. Free-text notes are the most sensitive and the messiest column in a CRM, they hold personal data, and an export for analysis rarely needs them (stripping personal data from a CSV before sharing it). If the analysis needs only a count of notes or their length, compute that at the source and leave the text behind. The DuckDB CSV options are documented in the CSV overview and the newline argument in Python's csv documentation.
A CSV import screen has one job: let a person with a spreadsheet get their data into your application without help. The job fails in a predictable way. The person uploads a file, something in it is not what the system expected, and the screen answers with \"Invalid file\" or with a red banner at the top of the page. They do not know which row, which column or what would work, so they try again, then email support. Every one of those cases can be prevented by a design that shows the person what the system understood and what it will do before it does it. This article describes that design, stage by stage, with the details that make the difference. It is the screen-level companion to CSV import and export features in applications, and the dry-run and error-report mechanics are in previewing a CSV upload before committing and reporting row-level errors back to the user.
\nThe first screen is where most errors are prevented. It has four elements, and each removes a class of mistake.
\nA template. A downloadable file with the exact header row and one or two example rows. Users who start from the template do not misspell headers, reorder columns or guess formats. Generate it from the same definitions that drive validation, so the two can never disagree, and give it a name that includes a version when the columns change.
\nThe rules, in words. Next to the upload field: which columns are required, what format dates and numbers use, which values are allowed for the fields with a fixed list, and the limits (file size, maximum rows). State them in the user's terms: \"Plan: free, pro or team\", \"Seats: a whole number, 1 or more\", \"Up to 10,000 rows per file\". A limit the user finds out about after a two-minute upload is a design bug (limiting file size and row count for CSV uploads covers the numbers).
\nWhat happens to existing records. A sentence that says whether the import adds new records only, or updates those it finds, and by what key: \"Customers are matched by email. Existing customers are updated; new ones are created.\" Without it, users do not know whether uploading is safe to repeat.
\nFormat choices. Accept CSV and, if your users live in spreadsheets, .xlsx, and say which encoding you read. If you only accept UTF-8, say so, and tell Excel users how to save as \"CSV UTF-8\".
After upload, do not start work. Show what the system found, and let the person correct it. The inspection step reports the delimiter it detected, the encoding, the number of rows and the header, in a sentence: \"We found 4 columns and 2,310 rows, separated by commas.\" If the detection is wrong, because a semicolon file was read as commas and every row has one column, the user sees it immediately and can choose the delimiter, which is better than a failure message after processing.
\nThen the mapping. A table of the file's columns on the left and the application's fields on the right, each with a dropdown, pre-filled by matching the headers (case, spacing, common aliases, as in choosing column names users recognize). The user confirms or corrects, and required fields without a match are flagged in red. A column in the file with no target is shown as \"ignored\", so the user knows it will not be imported, and a user who sees Notes marked ignored when they wanted it imported can fix it now and not after.
Remember what the mapping is for. A file with headers in French, a file with an extra leading column, a file with the columns in another order all work if the user can map them, and none works if the importer insists on exact names.
\nShow the first 10 to 20 rows as they will be understood, with the type each column was read as: the date shown as a date in the user's format, the number with its thousands separator, the text with leading and trailing spaces trimmed. Misreadings are visible at a glance in this view: a zip code shown as 2134 where the file has 02134, a date showing the wrong month, a column of names shifted into a column of cities by an unquoted comma. Showing parsed values is a far better test for the user than any message.
Then a summary of what Commit will do. In the test, a six-row file produced:
\ndry run: {\"created\":1,\"updated\":2,\"rejected\":3}On screen that is \"1 new customer, 2 customers will be updated, 3 rows have problems\" with a link to see the problems. This is the most important element on the page. It turns a vague \"import\" into three numbers a person can sanity-check against what they expected: if the file was meant to be 200 new customers and the summary says 200 updates, something is wrong, and the user finds out before anything has changed.
\nThe rows with problems should be listed in a form the user can act on. In the test, the report read:
\nline,column,value,problem\n4,plan,enterprise,\"plan must be one of: free, pro, team\"\n4,seats,,\"seats must be a whole number of 1 or more\"\n5,email,not-an-email,\"not a valid email address\"\n6,plan,gold,\"plan must be one of: free, pro, team\"Each line has the physical row in the user's file (with the header as line 1, so it matches the row number in their spreadsheet), the column using the user's own header name and not an internal field, the value that failed, and the rule. Two problems on one row, as on line 4, are both shown, so the user is not sent round the loop twice. Messages say what would work: \"must be one of: free, pro, team\" is a fix, and \"invalid value\" is a complaint. Avoid error text that blames (\"you entered an invalid email\"); the file is a dataset, and the message is a description of it.
\nFor many errors, show a summary first: \"128 rows have an unknown plan\" with five examples, and the full list as a download. A table of 128 individual lines hides the pattern that one cell of a column was misspelled throughout. The downloadable file should contain the original rows plus an error column, so that the user can fix and re-upload only those rows.
Some imports must be all-or-nothing, for instance a ledger that has to balance. Most do not, and refusing 2,300 good rows because 7 are bad is a design that makes people avoid the feature. Offer a choice, with the default that suits the data: \"Import the 2,303 valid rows and download the 7 that need fixing\" or \"Cancel and fix the file first\". Make the second the default only if partial import would leave the data inconsistent. If you import partially, make the next step obvious: the download of rejected rows is in the confirmation, not three clicks away.
\nEnd with a report, not a toast. The numbers (created, updated, rejected), a link to the rejected rows, a link to the records that changed if the application can filter by import, and an undo if you built one. Keep a list of past imports, with who ran them and when, so the person who comes back tomorrow and wonders what they did can see it. The record also serves support: when someone writes \"my import broke\", the import ID shows the file, the mapping and the errors.
\nA few details make the difference between usable and merely working. Keep the user's file name on every screen, so they remember which one they are importing. Do not time out silently: for a large file, show progress and let them leave the page. Make the whole flow work with a keyboard and a screen reader, which means real tables for the mapping and the preview, labels on every dropdown and error text that is not conveyed by color alone. Write for non-engineers: \"column\" and \"row\" are fine, \"delimiter\" needs a word of explanation, and \"constraint violation\" is not for users. And test the screen with a real person and a bad file, because the errors you did not think of are the ones that matter.
\nIf the import is rare, the file is produced by your own systems and the users are administrators who read the log, a plain form with a file field, a summary and a downloadable error report is enough, and the mapping stage is overhead. If the file always has the same layout because you control the template, skip mapping and validate the header strictly with a message that lists the expected and the found columns. Add stages as the audience widens: more users, more sources, more mistakes. The Mozilla Developer Network documents the browser side of file inputs in its file input guide, and the upload risks to design against are in OWASP's file upload cheat sheet.
","date_published":"2026-10-30T16:51:00+00:00","tags":["CSV"]},{"id":"https://datahacks.dev/library/csv-supplier-price-list-merged-header","url":"https://datahacks.dev/library/csv-supplier-price-list-merged-header","title":"A Supplier Price List With Merged Header Rows Fixed","summary":"A price list CSV has title lines, a header on row 4, category names written once per group and a footer. Skip, forward fill, filter and check the item count.","content_html":"Suppliers send price lists as spreadsheets and often as CSV exports of those spreadsheets, and a spreadsheet is a layout, not a table. There is a title on the first row, a validity note on the second, a blank row for air, then the header, then the data, in which each category name is written once on its first row and left blank beneath, as it would be in a merged cell. At the foot, a note about discounts. A person reads it perfectly. A parser sees a file whose first four rows are not data, whose category column is mostly empty and whose last rows are prose. The cleaning is three moves: skip the preamble, fill the blanks downward, and keep only rows that are items. This article does it on a small file in DuckDB and shows how to prove that no product was lost. It is one of the problems in worked CSV problems and the patterns behind them, and a sibling of parsing multi line headers in report style CSV files.
\nACME SUPPLY CO - PRICE LIST Q4,,,\nPrices exclude VAT. Valid until 31 Dec.,,,\n,,,\nCategory,Item,Unit,Price\nCables,USB-C 1 m,each,4.20\n,USB-C 2 m,each,5.80\n,HDMI 2 m,each,7.10\nAdapters,USB-C to HDMI,each,12.90\n,USB-A to USB-C,each,3.40\n,,,\nNotes: bulk discounts on request,,,Eleven lines. Lines 1 and 2 are title and terms, line 3 is blank, line 4 is the header, lines 5 to 9 hold five items under two categories, line 10 is blank and line 11 is a note. Every line has four comma-separated fields, because the spreadsheet had four columns, which is why a CSV reader does not complain: the file is perfectly regular as CSV and irregular as a table.
\nA naive read with the header on line one makes ACME SUPPLY CO - PRICE LIST Q4 the first column name and gives three unnamed columns. The right read skips the preamble, and the rest follows.
Skip the first four lines so that data starts at line 5, read with no header and explicit column names, and number the rows, since the fill-down needs a stable order:
\nWITH raw AS (\n SELECT row_number() OVER () AS n, *\n FROM read_csv('price_list.csv', header=false, skip=4,\n columns={'category':'VARCHAR','item':'VARCHAR','unit':'VARCHAR','price':'VARCHAR'},\n null_padding=true)\n)\nSELECT n,\n last_value(nullif(category, '') IGNORE NULLS) OVER (ORDER BY n) AS category,\n item, unit, price::DECIMAL(10,2) AS price\nFROM raw\nWHERE item <> '' AND item IS NOT NULL AND TRY_CAST(price AS DECIMAL(10,2)) IS NOT NULL\nORDER BY n;n category item unit price\n1 Cables USB-C 1 m each 4.20\n2 Cables USB-C 2 m each 5.80\n3 Cables HDMI 2 m each 7.10\n4 Adapters USB-C to HDMI each 12.90\n5 Adapters USB-A to USB-C each 3.40The last_value(... IGNORE NULLS) window carries the most recent non-NULL category down the rows: nullif(category, '') turns the empty strings of the blank cells into NULL, and IGNORE NULLS tells the window to skip them and return the last real value. Row 2 has no category of its own and gets Cables from row 1. Row 4 starts Adapters, and row 5 inherits it. The ORDER BY n is the contract: the fill is only correct if the rows are in file order, which is why they were numbered at the start. Without a number, a window with no order has no meaning, and a query that reorders rows would put categories on the wrong items.
The WHERE clause is the filter that drops what is not an item. It requires a non-empty item and a price that parses as a decimal. The blank line after the data has an empty item and is dropped. The note line, Notes: bulk discounts on request, has text in the first column, but it is the category column, and its item and price are empty, so it is dropped too. The filter does not name the footer. It describes what an item is, and anything else falls out, which is why the same query still works when the next issue has a different footer.
Three checks make the result trustworthy. The item count should equal the number you see in the source sheet. Here it is 5, and a person counting the rows in the original agrees. The category count should match as well: two categories, Cables and Adapters, each with the right number of items (3 and 2). And every kept row must have a parseable price, which the filter guarantees, so a different check is needed to catch the opposite error, a real item dropped because its price was written oddly. Count the rows that were excluded and look at them:
SELECT n, category, item, price FROM raw\nWHERE NOT (item <> '' AND item IS NOT NULL AND TRY_CAST(price AS DECIMAL(10,2)) IS NOT NULL);On the sample this returns the blank line and the note line, nothing else. If a row with an item and an odd price, say POA or 4,20 with a decimal comma, appears here, it is a product the filter dropped, and you want to know. The excluded-rows query is the part people skip, and it is the one that finds the missing product.
The skip=4 is fragile. If the supplier adds a line to the terms next quarter, the header moves to line 5 and the query silently reads the header as an item (and drops it, since its price does not parse) and shifts nothing else, so it may look fine. Do not hard-code the skip. Find the header by its content: read the file as one text column, locate the first line that starts with Category,, and use that line number:
SELECT min(n) AS header_line FROM (\n SELECT row_number() OVER () AS n, line\n FROM read_csv('price_list.csv', header=false, columns={'line':'VARCHAR'}, delim='\\x01', quote='', escape='')\n) WHERE line LIKE 'Category,%';Fail with a clear message if there is no such line, and pass the number into the skip option of the main read. A header that moves is acceptable, and a header that cannot be found stops the load, which is what you want.
Also decide what to do about the other regular changes: a new column added to the right (extra empty fields at the end of the title lines, and a new header name), a unit column that stops being filled, a second sheet exported as a second block below the first. Each is a reason the script should compare the header it found with the one it expects (handling a CSV provider who changes columns without warning).
\nThe category column is the usual merged cell. The same technique handles others: a supplier name in the first column that applies to the rows below, a date written once per day, a region on the first row of each block. The rule is the same, fill down with IGNORE NULLS, over rows in file order. Merged cells across columns, a header cell that spans three columns, need a fill to the right, which is a different problem and is covered in parsing multi line headers in report style CSV files.
Be careful about the case where a blank is meaningful. If a category cell is blank because the item belongs to no category, and not because it was merged, forward fill would assign it to the wrong group. The test is whether the first data row has a category: a blank in the first row means the layout is different from what you assumed, and the script should stop.
\nA supplier who changes the layout every issue is a supplier whose price list should come in a different form. Ask for a flat export, a single header and one item per row with the category in every row, or a data feed. If they cannot, keep the script small, keep a sample of each layout beside it as a test and make it fail on anything unfamiliar. A price list drives money decisions, and a parsing error that drops a product or shifts a price is expensive in a way that a failed job is not. The window function behind the fill is documented in DuckDB's window function reference, and the reader options in its CSV overview.
","date_published":"2026-10-30T16:51:00+00:00","tags":["CSV"]},{"id":"https://datahacks.dev/library/csv-log-export-unescaped-quotes","url":"https://datahacks.dev/library/csv-log-export-unescaped-quotes","title":"A Log Export With Unescaped Quotes and How to Read It","summary":"A log export has quotes inside quoted fields without doubling them. See what Python and DuckDB return, which setting reads them right and how to prove it.","content_html":"A web server or an application writes its log as CSV, and one of the columns is a request path or a user message, which can contain double quotes. The exporter wraps the field in quotes and does not double the ones inside, so the file has \"/search?q=\"red shoes\" size 9\" where RFC 4180 requires \"/search?q=\"\"red shoes\"\" size 9\". Standard parsers are written for the rule, and when it is broken they diverge: each reads the line in its own way, and none of the ways is an error. The result is fields that look plausible and are wrong. This article feeds the same four lines to Python and DuckDB, compares what comes back, finds the setting that reads them as intended and shows how to prove the repair. It is one of the problems in worked CSV problems and the patterns behind them, and the rules it breaks are in how CSV quoting works.
ts,method,path,status\n2026-10-02T10:00:01,GET,\"/search?q=red shoes\",200\n2026-10-02T10:00:04,GET,\"/search?q=\"red shoes\" size 9\",200\n2026-10-02T10:00:09,POST,\"/cart\",302\n2026-10-02T10:00:12,GET,\"/search?q=5\" nails\",404Lines 1 and 3 are fine: quoted fields with no quotes inside. Line 2 has a quoted field whose value contains \"red shoes\", and the exporter wrote the inner quotes bare. Line 4 has a quote that stands for inches, 5\", also bare. A human reads line 2 as /search?q=\"red shoes\" size 9 and line 4 as /search?q=5\" nails.
Python's csv module with its defaults reads a quote in the middle of a quoted field as the start of an escape, or as a literal, depending on the next character, and for these lines it returns:
['2026-10-02T10:00:01', 'GET', '/search?q=red shoes', '200']\n['2026-10-02T10:00:04', 'GET', '/search?q=red shoes\" size 9\"', '200']\n['2026-10-02T10:00:09', 'POST', '/cart', '302']\n['2026-10-02T10:00:12', 'GET', '/search?q=5 nails\"', '404']Each row has four fields, and the damaged ones are wrong: the quote that opened \"red shoes\" has vanished and reappeared after size 9, and the line 4 path lost its inch mark and gained a trailing quote. The field count is right and the content is not, which is the worst kind of error.
DuckDB's default settings sniff the file and, finding the quoting inconsistent, return the fields with the outer quotes kept as ordinary characters:
\n('2026-10-02T10:00:01', 'GET', '\"/search?q=red shoes\"', '200')\n('2026-10-02T10:00:04', 'GET', '\"/search?q=\"red shoes\" size 9\"', '200')\n('2026-10-02T10:00:09', 'POST', '\"/cart\"', '302')\n('2026-10-02T10:00:12', 'GET', '\"/search?q=5\" nails\"', '404')Now every path, including the good ones, has quotes around it. The sniffer decided that the quote character is not a quote, so line 1 and line 3 are damaged too. That is the sniffer's honest answer to a file that does not follow the rules, and it is a file-wide change of meaning, so it does more harm in this case than Python's mistake, which was confined to the bad lines.
\nDuckDB has a strict_mode option. With strict_mode=false, the reader allows quotes inside quoted fields and closes a field when the quote is followed by a delimiter or the end of the line, which is the rule a human applies to this file:
SELECT * FROM read_csv('weblog.csv', all_varchar=true, strict_mode=false);('2026-10-02T10:00:01', 'GET', '/search?q=red shoes', '200')\n('2026-10-02T10:00:04', 'GET', '/search?q=\"red shoes\" size 9', '200')\n('2026-10-02T10:00:09', 'POST', '/cart', '302')\n('2026-10-02T10:00:12', 'GET', '/search?q=5\" nails', '404')All four match the intended values: the good lines are unchanged, \"red shoes\" keeps both its quotes inside the path and the inch mark survives. Adding ignore_errors=true on top gave the same four rows, and it would also skip rows the reader cannot parse, which I did not need and which hides damage, so I left it off.
The rule that worked was that a quote ends a field only when the next character is a comma or a line break. That heuristic is exactly right for this file and it is still a heuristic: a legitimate field containing \", in its text, say a quoted phrase followed by a comma and a space, would be closed early. So the fix is not \"use lenient mode\" in general. It is \"use lenient mode for this exporter and prove that it reads this exporter's output correctly\".
Run three checks on the repaired read. Every row must have exactly four fields, which in DuckDB is true by construction of the schema and needs checking in a tool that does not enforce columns. The count of rows must equal the count of lines in the source minus the header, adjusted for embedded breaks, which here is 4. And the broken lines must read as expected, which you check by eye: pull out the rows whose path contains a quote character and look at them.
\nSELECT path FROM read_csv('weblog.csv', all_varchar=true, strict_mode=false) WHERE path LIKE '%\"%';/search?q=\"red shoes\" size 9\n/search?q=5\" nailsTwo rows, both as intended. On a real log with millions of lines, that query might return thousands, and reading a hundred of them tells you whether the heuristic holds. If any of them looks cut off or merged with a neighbor, the heuristic fails for some shape of line you have not thought of.
\nA further check catches the dangerous cases: look for fields that end with a comma followed by something that looks like the next column, and for rows where a later column, the status, is not a three-digit number. A mis-split line puts text in a numeric column. A numeric cast on the status column (TRY_CAST(status AS INTEGER)) with a count of failures is a cheap alarm.
Other parsers have similar escape hatches. Python's csv module accepts strict=True to raise on malformed input instead of guessing, which is better for finding the bad lines and does not repair them, and quoting=csv.QUOTE_NONE with a custom quote character turns quoting off, which breaks fields with delimiters. pandas has on_bad_lines for skipping or warning. A regular expression that implements the \"quote ends before a comma or end of line\" rule is a handful of lines and works on any language, at the cost of being your code to maintain:
import re\nFIELD = re.compile(r'\"((?:[^\"]|\"(?!,|$))*)\"(?:,|$)|([^,]*)(?:,|$)')\n\ndef split_line(line):\n out, pos = [], 0\n while pos < len(line):\n m = FIELD.match(line, pos)\n out.append(m.group(1) if m.group(1) is not None else m.group(2))\n pos = m.end()\n return outRun on the four lines, it returned the same values as DuckDB's lenient mode:
\n['2026-10-02T10:00:04', 'GET', '/search?q=\"red shoes\" size 9', '200']\n['2026-10-02T10:00:12', 'GET', '/search?q=5\" nails', '404']That pattern reads a quoted field as a run of characters where a quote is allowed if it is not followed by a comma or the end, which is the same heuristic. Test it on your broken lines, include a line with \", inside a field in the tests, and be ready to find that it does not handle it; I did not try that case. For anything beyond a one-off, prefer the parser option and keep the regex as a cross-check.
Some damage is not recoverable. If the quoted field contains both an unescaped quote and the delimiter, such as \"/search?q=5\", 6\", there are two readings and the text itself does not choose. If a log line is truncated in the middle of a quoted field, there is no closing quote to find. If a field contains a newline and an unescaped quote, the record boundary is lost. For these, count the rows that fail the field-count and status checks and send them to a rejects file with their line numbers (logging rejected rows from a CSV import for review), and tell the owner of the exporter.
The real fix is at the source: configure or patch the log writer to double quotes inside quoted fields, or switch the format to one that has no such ambiguity, such as tab-separated with escaping or JSON lines. RFC 4180 states the rule in one sentence, and DuckDB documents the reader options in its CSV overview and its page on reading faulty CSV files.
","date_published":"2026-10-30T16:51:00+00:00","tags":["CSV"]},{"id":"https://datahacks.dev/library/csv-government-dataset-footnotes-data","url":"https://datahacks.dev/library/csv-government-dataset-footnotes-data","title":"A Government Dataset With Footnotes in the Data Cells","summary":"Statistical CSVs put flags in value cells: 1,204,300 (p), 5.3 e, and dots for missing. Split number from flag, keep both and check against the stated total.","content_html":"Official statistics are published with care and with symbols. A population figure is 1,204,300 (p) because it is provisional. An unemployment rate is 5.3 e because it is an estimate. A cell holds .. or x because the value is unavailable or was withheld to protect someone's privacy. Each of those marks is information that a statistician wants beside the number, and a CSV has one place to put it, the cell itself. So the file is exactly what a person needs and almost useless to a program: the column that should be numeric holds text, with flags in various positions and several tokens for missing. This article shows how to split the number from its flag, keep both, check the result against a stated total and leave the suppressed cell alone. It is one of the problems in worked CSV problems and the patterns behind them, and it shares a technique with an analytics export with percent signs and dashes.
region,year,population,unemployment_rate,note\nNorth,2024,\"1,204,300 (p)\",4.1,\nSouth,2024,\"980,550\",5.3 e,\nEast,2024,..,x,suppressed\nWest,2024,\"1,150,200 r\",4.8,revised\nTotal,2024,\"4,100,000 (p)\",4.7 e,Four things are going on. The population has thousands separators and, in three rows, a suffix flag: (p), r. The unemployment rate has a flag too, a bare e after a space. East has .. for population and x for the rate, tokens that mean \"no value\" and that a statistical agency would explain in a legend. And there is a Total row mixed with the regions, which is both a trap (summing the column double counts) and an opportunity (it is a check).
The legend matters. The meaning of each symbol belongs to the publisher, and the file does not carry it. Before you write a rule, find the key in the documentation of the dataset and write it into your code: (p) provisional, e estimated, r revised, .. not available, x suppressed. Do not guess.
The value is a number at the start of the cell and a flag after it, and a regular expression separates them. Digits, commas and a decimal point at the start are the number, and whatever remains after trimming is the flag:
\nCREATE TABLE tidy AS\nSELECT region, year::INT AS year,\n TRY_CAST(replace(regexp_extract(population, '^[0-9,]+', 0), ',', '') AS BIGINT) AS population,\n nullif(trim(regexp_replace(population, '^[0-9,]+', '')), '') AS population_flag,\n TRY_CAST(regexp_extract(unemployment_rate, '^[0-9.]+', 0) AS DECIMAL(5,2)) AS unemployment_rate,\n nullif(trim(regexp_replace(unemployment_rate, '^[0-9.]+', '')), '') AS unemployment_flag\nFROM read_csv('gov.csv', all_varchar=true);region year population population_flag unemployment_rate unemployment_flag\nEast 2024 NULL .. NULL x\nNorth 2024 1204300 (p) 4.10 NULL\nSouth 2024 980550 NULL 5.30 e\nTotal 2024 4100000 (p) 4.70 e\nWest 2024 1150200 r 4.80 NULLRead it cell by cell. 1,204,300 (p) became 1204300 and the flag (p). The plain 980,550 has a number and no flag. 5.3 e became 5.30 with flag e. East's .. has no leading digits, so the number is NULL, and the flag column holds the token itself, .., which keeps the reason the value is missing: .. and x stay visible instead of vanishing into a NULL. That is the point of keeping a flag column. A NULL says nothing, and .. says \"not available\" per the legend.
The regex has to be written with care. '^[0-9,]+' takes a run of digits and commas, and it would also take a lone comma; for a field that might start with a comma, tighten it to '^[0-9][0-9,]*'. The decimal rates use '^[0-9.]+'. Negative values, a leading minus or parentheses, and a currency symbol are not in this file and would each need their own clause, as in an analytics export with percent signs and dashes. And since a missing flag is NULL, a statement like population_flag = '(p)' is false for a NULL, which is the behavior you want and a trap for a NOT IN filter.
A table of the distinct flags and how many rows carry each is the proof that you know all the symbols in the file:
\nSELECT population_flag, count(*) FROM tidy GROUP BY 1 ORDER BY 1;population_flag count\n(p) 2\n.. 1\nr 1\nNULL 1Compare the list with the legend. A flag that is not in the legend, a k or a *, means either the legend is incomplete or the cleaning has a bug, and you want to stop and find out. Put the allowed flags in a table and fail if a row carries something else. For files that arrive regularly, a new flag is an event: the publisher added a category, and the right response is to read what it means.
The Total row is a gift. A published total lets you test the parts. Sum the regions that have values and compare:
SELECT sum(population) FILTER (WHERE region <> 'Total') AS sum_regions,\n max(population) FILTER (WHERE region = 'Total') AS stated_total FROM tidy;sum_regions stated_total\n3335050 4100000The three regions with values add to 3,335,050, and the stated total is 4,100,000. They do not match, and the difference, 764,950, is a number worth understanding. In this file the explanation is the suppressed East row: the total includes a value that the table does not show. If exactly one cell is missing, the gap is its value. If two are missing, the gap is their sum, and you cannot separate them. If the numbers had matched, the check would have confirmed that the cleaning kept every value. If they differ by something that is not a suppressed cell, a row was dropped, or a flag swallowed a digit, or the total is computed on a different basis (rounded components, a different date). Always exclude the total row from sums, and write the filter, since a column sum that includes it is exactly double.
\nThere is a caution about what you do with the gap. Statistical agencies suppress cells because publishing them could identify a person or a firm, and a table that gives the total and all but one cell publishes the last one by subtraction. If you derive 764,950 for East and publish it, you may undo the protection that the source applied on purpose. Use the arithmetic as a check, and do not republish a derived value for a cell the source chose to hide. For small counts, this is a matter of privacy and sometimes law (stripping personal data from a CSV before sharing it covers the neighboring ideas).
\nIt is tempting to drop the flag columns once the numbers are clean. Resist. A provisional figure may be revised, an estimate has a margin the final figure does not, and a chart that treats them alike overstates how much is known. Keep the flags, show them where they matter (a footnote mark, a lighter bar) and filter on them when you need final data only: WHERE population_flag IS NULL selects the values with no qualifier. When a later release replaces (p) with the final number, compare: a provisional value that moves by more than a tiny amount is itself a finding.
Some marks change the meaning of the number. A superscript or letter that says the figure refers to a different population or a different definition, a break in series note, a unit change from thousands to millions halfway down the table: those are not decoration. If the legend says a mark changes the basis, carry it as a column and do not combine rows with different marks in one aggregate. And when the file carries footnotes as separate lines at the bottom (1. Excludes...), read them before you trust the table, because they often hold the definitions. The format gives you none of this, and RFC 4180 is silent on it. For the DuckDB functions used, see its regular expression functions and CSV overview.
Survey platforms export one row per respondent and one column per question. With ten questions that is a fine table. With three hundred, it is a spreadsheet nobody can read, a database table that is awkward to query, and a file where adding a question changes the schema. Analysis wants the other shape: one row per answer, with the respondent, the question and the value, so that \"average by question\" is a GROUP BY and \"answers by country\" is a join. Turning wide into long is one of the most useful moves in data wrangling, and in DuckDB it takes a single statement that does not list the columns. This article does it on a 302-column file, shows the proof that no answer was lost and covers the question metadata that makes the result readable. It is one of the problems in worked CSV problems and the patterns behind them.
The first columns identify the respondent, and the rest are the questions. A trimmed view of the header and a row:
\nrespondent,country,Q1,Q2,Q3,Q4,Q5,...,Q299,Q300\nR001,DE,3,,1,5,,...,2,4(The ... stands for the columns elided here; the file has all 300.) Values are answers on a one-to-five scale, and an empty cell means the question was not asked, was skipped or was not shown because of a branch. DuckDB reports 302 columns for the file, which is what a spreadsheet would show after scrolling for a while.
Read it as text first, since an empty column of answers can be inferred as a number or a boolean depending on what the sniffer sees, and you will cast on purpose after the reshape:
\nCREATE TABLE wide AS SELECT * FROM read_csv('survey.csv', all_varchar=true);UNPIVOT turns columns into rows. The old way lists every column to unpivot, which for 300 columns is a long and fragile statement. DuckDB lets you select the columns with a regular expression through COLUMNS:
CREATE TABLE long_t AS\nSELECT respondent, country, question, answer::INTEGER AS answer\nFROM (UNPIVOT wide ON COLUMNS('^Q\\d+$') INTO NAME question VALUE answer)\nWHERE answer <> '';The pattern ^Q\\d+$ selects columns named Q followed by digits, which excludes respondent and country. INTO NAME question VALUE answer names the two new columns: the old column name goes into question and the cell goes into answer. The columns not mentioned, the respondent and the country, are kept on every output row. The WHERE clause drops blank cells, and the cast to integer happens after the unpivot, so a non-numeric answer would raise an error that points at a value instead of making the column text.
A new question in next month's export, Q301, is picked up by the pattern with no change to the statement, which is the main practical gain over listing the columns. The converse is a hazard: a column that matches the pattern but is not a question, such as a Q4_comment free-text field, would not match ^Q\\d+$ and be left out, and one called Q99 that holds a comment would be pulled in and fail the integer cast, which is the behavior you want. Decide the pattern from the platform's naming scheme and check what it selects: SELECT column_name FROM (DESCRIBE wide) WHERE regexp_matches(column_name, '^Q\\d+$') lists them.
The long table should hold exactly the non-blank question cells of the original. Compute that count in the wide table without unpivoting, and compare:
\nSELECT count(*) AS answers, count(DISTINCT respondent) AS people, count(DISTINCT question) AS questions\nFROM long_t;answers people questions\n1057 5 2991,057 answers from 5 people across 299 distinct questions. Five respondents by 300 questions is 1,500 cells, and about 70 percent were filled. To verify 1,057 independently, count in the original, for example with a script that reads the file with the csv module and counts non-empty cells in the question columns, or with a SQL expression that sums (Q1 <> '')::INT + (Q2 <> '')::INT ... generated for all 300 columns. The two numbers must agree. If they do not, the pattern missed a column, a cell held whitespace that <> '' did not catch (trim it first), or a header was duplicated.
The 299 is also a result: one of the 300 questions had no answers from any of the five respondents. In a real survey, that is worth knowing, as a question nobody answered may be a branch no one reached or a broken form.
\nThe reshape turns wide questions into ordinary SQL. Answers per question and the mean for the most-answered ones:
\nSELECT question, count(*) AS n, round(avg(answer), 2) AS mean\nFROM long_t GROUP BY question ORDER BY n DESC, question LIMIT 3;question n mean\nQ102 5 3.0\nQ105 5 1.8\nQ106 5 1.8And by group, joining nothing: the respondent's country travels with every answer, so the breakdown is a group by:
\nSELECT country, round(avg(answer), 2) AS mean_answer, count(*) AS answers\nFROM long_t GROUP BY country ORDER BY country;country mean_answer answers\nDE 3.10 213\nFR 2.89 228\nGB 3.00 616Those figures are from random data and mean nothing. What matters is the form: one GROUP BY, no column names in the query. Note the counts, since GB has 616 answers because three respondents were in GB. A mean over 213 answers from one person is not 213 independent observations. If you want to compare countries, the unit is the respondent, so average each respondent first and then compare the respondents' averages, or the analysis overstates its certainty (comparing two time periods in a CSV makes the same point about bases).
Q102 means nothing to a reader. Keep the question text, the scale and the group in a small table, usually exported from the survey tool as a separate file or as the second header row of the export:
question,text,scale,topic\nQ102,How satisfied are you with delivery?,1-5,delivery\nQ105,Would you recommend us?,1-5,loyaltyJoin it to the long table, and every query gains words. A second header row of question text in the export itself, common in survey tools, is the multi-line header problem, and skip plus a separate read of that row gives the text (parsing multi line headers in report style CSV files). Also separate scale types: a 1-to-5 rating, a yes/no, a multiple-choice code and a free-text answer cannot share one answer column without casting, so either keep answer as text and cast per question type using the metadata, or split the long table by type. Multiple-select questions often arrive as several columns (Q40_1, Q40_2, Q40_3, one per option), which the long form turns into question = 'Q40_2' rows; split the name into a question and an option with string_split if you need to group them.
Do not unpivot everything by reflex. A wide table suits tools that want a feature matrix, such as a statistics package or a model, where each question is a variable and each respondent a case: reading the long form back into a matrix is another pivot. It also suits questions you will always analyze together, a battery of ten items that make a scale, where the mean across the items per respondent is a one-line expression. And a 302-column table is fine to store in Parquet or a database, where columns are cheap, even if it is miserable to read in a spreadsheet.
\nUse the long form for exploring, counting and joining, and the wide form for modeling and for the export you give back. Keep both reproducible from the raw file with a script. For very large survey files, convert once to Parquet to avoid reparsing 300 columns each time (why Parquet beats CSV for repeated analysis). The statement is documented in DuckDB's UNPIVOT reference, and its CSV overview covers reading wide files.
","date_published":"2026-10-30T16:51:00+00:00","tags":["CSV"]},{"id":"https://datahacks.dev/library/merging-yaml-files-command-line","url":"https://datahacks.dev/library/merging-yaml-files-command-line","title":"Merging YAML Files From the Command Line With yq Safely","summary":"Deep-merge a base and an override with yq eval-all, choose whether lists replace or append, and check what a null or an empty value in the override does.","content_html":"Layered configuration is common: a base file with defaults, a second file with overrides for an environment. In Python a deep merge is a ten-line function. On the command line, in a CI step or a deployment script, yq does it in one expression. This post shows the expression, what each operator does to nested mappings and lists, and the traps around nulls, ordering and writing the result, tested on a base and an override file.
server:\n host: db1\n port: 5432\n opts: {ssl: true, pool: 5}\nfeatures: [search, export]server:\n host: db2\n opts: {pool: 20}\nfeatures: [billing]The override changes the host and the pool size and does not mention the port or ssl. It also supplies a different features list.
yq eval-all 'select(fi==0) * select(fi==1)' a.yaml b.yamlserver:\n host: db2\n port: 5432\n opts: {ssl: true, pool: 20}\nfeatures: [billing]eval-all (alias ea) reads all the files into one stream of documents before running the expression, unlike plain eval, which runs the expression on each file separately. fi is the file index, starting at 0, so select(fi==0) is the first file and select(fi==1) the second. The * operator is the deep merge: where both sides have a mapping at the same key it recurses, and where they differ the right side wins. The result keeps port: 5432 and ssl: true from the base, takes host: db2 and pool: 20 from the override. A shallow merge would have kept only the override's server block.
The features list was replaced by the override's list, [billing]. That is the default behavior for arrays with *.
The variant *+ appends arrays:
yq eval-all 'select(fi==0) *+ select(fi==1)' a.yaml b.yamlserver:\n host: db2\n port: 5432\n opts: {ssl: true, pool: 20}\nfeatures: [search, export, billing]Same mappings, and the list now holds all three. Other modifiers exist in the yq manual for specific needs: *d to merge arrays by index (deep), *n to only add keys that do not exist in the base (so the base wins), and *? to merge only keys that exist. Pick the one whose meaning matches your policy and write it in a comment next to the command. The point is the same as in the Python function: the list policy is a decision, and the operator is where it lives. Replacing is the usual choice for configuration where the override means \"exactly these\", and appending suits lists of things to run or paths to search.
yq prints the merged document to standard output, so redirect it to a new file:
yq eval-all 'select(fi==0) * select(fi==1)' base.yaml prod.yaml > merged.yamlDo not use -i with several input files. In a test, yq -i '...' f1.yaml f2.yaml f3.yaml rewrote only the first file, replacing its content with the concatenation of all three as a multi-document stream, and left the other two unchanged (using yq to update many YAML files at once). For a merge, write to a new path, validate it and move it into place.
To merge more than two files, list them in order. Later files override earlier ones, and the expression reduces across them:
\nyq eval-all '. as $item ireduce ({}; . * $item)' base.yaml env.yaml local.yamlThe ireduce form folds a list of any length with the * operator. The simple select(fi==0) * select(fi==1) form works for exactly two files.
A null in the override is a value like any other. If the override has debug: ~, the merged debug is null, replacing the base's false. Whether that is what you want is the same question as in Python: is a null \"unset\" or \"no change\"? yq has an operator to ignore nulls in some forms, and the simplest protection is to not write empty keys in override files. Lint for them with yamllint or a schema.
A key with an empty mapping (opts: {}) in the override merges as no change to the base's opts, since a deep merge of an empty mapping adds nothing, while a key with opts: ~ replaces it. The two look similar in a file and behave differently (YAML null values and the empty string question).
yq keeps the comments of the inputs where it can, and the merged layout follows the first file with the second's new keys added at the end of their mappings. Flow-style mappings from the input (opts: {ssl: true, pool: 20}) stay in flow style. Run a formatter on the result if you commit it, so it matches the repository style. For merged files that are generated at deploy time and never committed, the layout does not matter.
Load the merged file with the parser the application uses and print it. A merge can succeed and produce a value that the parser reads differently.
Validate against the schema. Unknown keys from a typo in the override pass through the merge untouched, and a schema with forbidden extras reports them (validating config YAML with JSON Schema).
Check for duplicate keys in each input first. A duplicate in the base is resolved by the parser before the merge, which hides it.
Diff the merged file against the base to review what the override actually changed: diff base.yaml merged.yaml.
Keep the layer order in the command, not in file naming alone, so it is visible in the script.
A common use is composing the effective configuration at deploy time. The repository holds base.yaml and one file per environment. The deploy script merges them into a temporary file, validates it, and mounts or uploads it:
yq eval-all '. as $item ireduce ({}; . * $item)' base.yaml \"env/$ENVIRONMENT.yaml\" > /tmp/effective.yaml\ncheck-jsonschema --schemafile schema.json /tmp/effective.yamlI ran the folding form on three files and the result combined all three layers: server took host: db2 from the second file and port: 1 from the third, with opts merged from the first two, and the extra key from the third appeared at the end. The effective file is an artifact, so do not commit it. Print it in the deploy log with secrets masked, because the first question in any incident is what configuration was actually applied.
Keep the base free of environment-specific values and the overrides small. An override file that repeats most of the base has stopped being an override, and the next change to the base has to be made in several places. The merge makes small overrides cheap, so use it that way.
\nThe cluster hub is working with YAML from the terminal. The same merge in code, with the policy discussion, is in merging two YAML files in Python with a deep merge, and layering by environment in environment specific YAML config without copy and paste. The operators are in the yq multiply and merge documentation, and the file-index variable in the yq fi reference. The merge semantics follow the YAML 1.2.2 mapping rules. Compare two inputs in the YAML formatter.
Lists of records need to merge by key. Combining two lists of services by name is not what * or *+ does. Use a script, or yq's unique_by and group_by operators, with care.
The platform has its own overlay mechanism. Kustomize and Helm merge Kubernetes manifests with knowledge of the fields. Use them for those files.
\nThe merge needs conditionals. Logic belongs in code with tests.
\nYou must preserve exact formatting. Merge, then run the formatter your repository uses, or accept normalized output.
","date_published":"2026-10-30T16:51:00+00:00","tags":["YAML"]},{"id":"https://datahacks.dev/library/validating-yaml-configuration-pydantic","url":"https://datahacks.dev/library/validating-yaml-configuration-pydantic","title":"Validating YAML Configuration With pydantic Models","summary":"Load YAML, validate it with pydantic models and get every error with its path. The Norway problem caught, extra keys rejected and ranges enforced.","content_html":"Loading a YAML file tells you it is YAML. It does not tell you it is the configuration your program needs. A misspelled key is ignored, a port written as a word survives until it is used, a boolean where a string belongs passes through. The cheapest fix is a schema checked at load time, and in Python the most convenient way to write one is a pydantic model. This post builds one for a small configuration, runs it on a good and a bad file, and shows the errors it produces.
\nA configuration with an environment name, a list of servers and a timeout:
\nfrom typing import Literal\nfrom pydantic import BaseModel, ConfigDict, Field\n\nclass Server(BaseModel):\n model_config = ConfigDict(extra=\"forbid\")\n host: str\n port: int = Field(ge=1, le=65535)\n tls: bool = False\n\nclass Config(BaseModel):\n model_config = ConfigDict(extra=\"forbid\")\n env: Literal[\"dev\", \"staging\", \"prod\"]\n servers: list[Server]\n timeout: float = 30.0The class bodies read like a specification. env must be one of three strings. servers is a list of Server, each with a string host, an integer port between 1 and 65535 and an optional boolean tls. timeout is a number with a default. extra=\"forbid\" makes any key not declared an error, in both models.
env: prod\nservers:\n - {host: a.example, port: 443, tls: true}\ntimeout: 12Config.model_validate(yaml.safe_load(good))env='prod' servers=[Server(host='a.example', port=443, tls=True)] timeout=12.0The result is a typed object, config.servers[0].port is an int, and timeout was converted from the integer 12 to the float 12.0. Code downstream can rely on the types. (pydantic converts compatible values by default, which is called lax mode; ConfigDict(strict=True) switches it off when you want 12 rejected for a float field.)
env: production\nservers:\n - {host: a.example, port: \"https\"}\n - {host: b.example, port: 70000, tls: maybe, extra_key: 1}\ntimeout: soon\nretries: 3Validating it:
\ntry:\n Config.model_validate(yaml.safe_load(bad))\nexcept ValidationError as e:\n print(e.error_count(), \"errors\")\n for err in e.errors():\n print(\" \", \".join(str(p) for p in err[\"loc\"]), \"->\", err[\"msg\"])7 errors\n env -> Input should be 'dev', 'staging' or 'prod'\n servers.0.port -> Input should be a valid integer, unable to parse string as an integer\n servers.1.port -> Input should be less than or equal to 65535\n servers.1.tls -> Input should be a valid boolean, unable to interpret input\n servers.1.extra_key -> Extra inputs are not permitted\n timeout -> Input should be a valid number, unable to parse string as a number\n retries -> Extra inputs are not permittedSeven errors from one run. That is the main advantage over writing checks by hand: pydantic collects every error instead of stopping at the first, and each one has a location path (servers.1.port is the port of the second server) and a plain message. That output can go straight into an error message for the person who edited the file. The misspelled-key class of mistakes shows up as \"Extra inputs are not permitted\", which is what extra=\"forbid\" is for. Without it, extra_key and retries would have been silently ignored, and the typo tsl: true would have left TLS off.
The type declarations catch the surprises that YAML's type guessing introduces. A country code field:
\nclass C(BaseModel):\n country: str\nC.model_validate(yaml.safe_load(\"country: NO\"))country: NO -> Input should be a valid string | input: FalsePyYAML loaded the unquoted NO as the boolean False; the field is declared str; pydantic rejected it and reported the input it received, False. The error is a loud, early signal that the file needs quotes (the Norway problem and YAML boolean surprises). Compare the failure mode without validation: a False flowing into code that expected a code, found when a customer in Norway cannot be selected.
The same applies to versions loaded as floats, identifiers with leading zeros loaded as numbers and times loaded as integers. Declare them str, and the loader tells you. By default pydantic's lax mode does not convert numbers to strings, so a str field given 3.1 for a version written 3.10 is rejected, which is the behavior you want here (YAML version numbers turning into floats).
Defaults and optional fields. timeout: float = 30.0 makes the key optional. Use str | None = None for fields that may be absent or null.
Constraints. Field(ge=1, le=65535), min_length, pattern for strings, Literal[...] for enumerations. Put the constraints in the model, and the error messages describe them.
Custom validators. For rules involving several fields (a tls server must have port 443), use @model_validator. For parsing special strings (\"30s\" into seconds), use @field_validator.
Nested models and lists validate recursively, with paths that show where.
\nSecrets. SecretStr hides a value in logs and error output, which matters for passwords in configuration.
Environment overrides. The companion package pydantic-settings reads values from environment variables with the same models, which makes layering file and environment values straightforward (environment variable substitution in YAML config).
Validate once, at the boundary where the file is loaded, and pass the typed object to the rest of the program. Put the load function and the model in one module that the whole application imports:
\ndef load_config(path) -> Config:\n with open(path, encoding=\"utf-8\") as f:\n data = yaml.safe_load(f) or {}\n return Config.model_validate(data)A failing load exits with the full list of errors at start-up, not hours later in a code path nobody exercised. In CI, add a test that loads the example and production configuration files, so a bad edit fails the pull request. The same model can generate a JSON Schema (Config.model_json_schema()) for editors and for tools that validate outside Python (validating config YAML with JSON Schema).
The error list is the raw material for a good message, and a little formatting makes it better. Print the file name, then one line per error with the path in a form the author recognizes (servers[1].port) and the message. Add the received value when it is short, which pydantic includes in each error as input; for a bad port, \"got 70000\" is more useful than \"less than or equal to 65535\" alone. Group errors by top-level section when the list is long. And exit with a nonzero status so scripts and CI notice.
If your users want line numbers, there are two routes. Compose the YAML into a node tree with yaml.compose, which keeps the start mark of every node, and look up the path from each error to find the line. Or load with ruamel.yaml round-trip mode, whose containers expose lc.line and lc.col for keys and items. Either costs thirty lines and turns \"servers.1.port is invalid\" into \"line 7, column 30: port is invalid\", which is worth it for files that non-programmers edit.
A last tip is to validate example files in the documentation. Keep an example.yaml that you ship in the repository and load it in a test with the model. When someone adds a field to the model and forgets the example, the test fails, and the example stays truthful.
The cluster hub is reading and writing YAML in Python safely. The loading step is in loading YAML in Python with PyYAML safe_load. The model features are documented in the pydantic documentation and its validation errors reference. Check how a value loads in the YAML to JSON tool.
\nOther languages read the file. A Python model validates for Python. Export the JSON Schema and validate in CI for every consumer.
\nThe file is huge. Validation of thousands of records is fast in pydantic 2 and still not the right tool for bulk data.
\nThe constraints are external. A port that must be free, a host that must resolve and a file that must exist are checks against the world. Do them in a separate start-up step with clear messages.
\nYou need the original text. Validated objects have lost comments and line numbers, so errors cannot point to a line. If line numbers matter to your users, validate the loaded nodes with a library that tracks marks, or map paths to lines by composing the YAML tree.
","date_published":"2026-10-30T16:51:00+00:00","tags":["YAML"]}]}