Converting CSV to XML With a Short and Safe Python Script

XML is more demanding than JSON about what text may appear and what names may be. A CSV header is any string at all, and a CSV field can contain anything the encoding allows. XML element names have to follow a grammar, and the text inside them cannot contain most control characters, and the usual conversion script from a tutorial does not check either. This article converts a CSV to XML with Python's standard library, finds the two places where it produces output that no parser will accept, and shows how to write a large file without building it in memory.

The obvious version

Map the file to a document: the root element is the file, each row is an element, each column is a child element. With ElementTree:

SRC = "/batches/csv-0001-0200/lab/data/conv/people.csv"
rows = list(csv.DictReader(open(SRC, newline="", encoding="utf-8")))

print("1. The naive version: header names become tags")
root = ET.Element("people")
for r in rows[:2]:
    e = ET.SubElement(root, "person")
    for k, v in r.items():
        ET.SubElement(e, k).text = v
print(ET.tostring(root, encoding="unicode")[:190], "...")

python lab/conv/t0147.py

1. The naive version: header names become tags
<people><person><id>1</id><name>Ana Silva</name><zip>02134</zip><score>12.50</score><active>true</active><joined>2025-03-01</joined><note>plain</note></person><person><id>2</id><name>O'Neil, ...

2. A header that is not a valid XML name
written:  <row><first name>Ana</first name></row>
parsed:   ParseError: not well-formed (invalid token): line 1, column 16

3. Tag names made safe: ['first_name', '_2fa', 'e-mail', 'Zo_', 'a_b']

4. Escaping of the awkward row (ElementTree does it)
<person><note>&lt;b&gt;bold&lt;/b&gt; a|b &amp; "q"</note></person>
<person note="line one&#10;line two" name="Bo &quot;Bee&quot; Chen" />
attribute round trip: 'line one\nline two'

5. A control character in the data
written: '<note>bell\x07here</note>'
parsed:  ParseError: not well-formed (invalid token): line 1, column 10
cleaned: 'bellhere'

The first block of the output is the result for two rows: <people><person><id>1</id><name>Ana Silva</name><zip>02134</zip>.... It has all the right pieces, and the zip keeps its zero, since XML text is text. Which of the choices in it are yours to make? Elements or attributes: I put values in child elements, because attributes cannot hold a newline readably and cannot repeat. Empty fields: an empty string becomes <note></note> or <note/>, and it is the same thing to a parser. And the element names: they came from the header, which is where the first problem is.

A header is not always a name

XML names start with a letter or an underscore and then contain letters, digits, hyphens, underscores and periods (the Names production allows more, including many non-ASCII letters). A column called first name, 2fa or a/b is a perfectly good CSV header and not a valid element name. The second block of output:

print("\n2. A header that is not a valid XML name")
bad = ET.Element("row")
ET.SubElement(bad, "first name").text = "Ana"
text = ET.tostring(bad, encoding="unicode")
print("written: ", text)
try:
    ET.fromstring(text)
except ET.ParseError as e:
    print("parsed:   ParseError:", e)

ElementTree accepts any string as a tag when it writes, so it wrote <row><first name>Ana</first name></row>. Parsing that back failed with ParseError: not well-formed (invalid token): line 1, column 16. The writer produced a file that its own reader rejects, and nothing complained at write time. A converter that you point at files you do not control needs a name step. This one replaces every run of unsupported characters with an underscore and prefixes an underscore when the first character is not a letter:

NAME = re.compile(r"[^A-Za-z0-9_.-]+")


def tag(name):
    t = NAME.sub("_", name.strip())
    return t if re.match(r"[A-Za-z_]", t) else "_" + t


print("\n3. Tag names made safe:", [tag(n) for n in ["first name", "2fa", "e-mail", "Zoë", "a/b"]])

Block 3 shows first name became first_name, 2fa became _2fa, e-mail stayed, and a/b became a_b. It also turned Zoë into Zo_, because the pattern is ASCII-only: XML permits ë in a name, and my function is stricter than it needs to be. That is the safe direction. The other risk, two different headers mapping to the same name (a b and a_b), needs a check for duplicates after the mapping. If the header names matter to the consumer, ask for an agreed list instead of deriving them, as the contract does.

Escaping is done for you, mostly

The writer escapes the characters that the markup uses:

print("\n4. Escaping of the awkward row (ElementTree does it)")
e = ET.Element("person")
ET.SubElement(e, "note").text = rows[8]["note"]
print(ET.tostring(e, encoding="unicode"))
e2 = ET.Element("person", {"note": rows[3]["note"], "name": rows[2]["name"]})
print(ET.tostring(e2, encoding="unicode"))
print("attribute round trip:", repr(ET.fromstring(ET.tostring(e2, encoding="unicode")).get("note")))

Block 4: the note <b>bold</b> a|b & "q" was written as <b>bold</b> a|b & "q". The angle brackets and the ampersand are escaped, and the double quotes are left alone in element text. In an attribute, quotes are escaped too (name="Bo "Bee" Chen"), and a newline inside an attribute was written as 
. Reading the attribute back returned 'line one\nline two', so the newline survived, which is not guaranteed by every XML library and is a reason to prefer elements for free text. Never build XML with string formatting for this reason: f"<note>{value}</note>" is correct for exactly the values that do not need escaping.

Control characters

This is the failure that shows up in production, months after the script was written. XML 1.0 allows tab, line feed and carriage return among the control characters and forbids the rest (see the Char production). The fifth block:

print("\n5. A control character in the data")
c = ET.Element("note"); c.text = "bell\x07here"
x = ET.tostring(c, encoding="unicode")
print("written:", repr(x))
try:
    ET.fromstring(x)
except ET.ParseError as err:
    print("parsed:  ParseError:", err)

INVALID = re.compile("[\x00-\x08\x0b\x0c\x0e-\x1f\ufffe\uffff]")
print("cleaned:", repr(INVALID.sub("", "bell\x07here")))

A text with a bell character (\x07) was written as is, <note>bell\x07here</note>, and parsing it back failed with not well-formed (invalid token): line 1, column 10. Real CSV files have control characters in them: a vertical tab from a copied cell, a form feed, a stray NUL, a backspace. The fix is to decide what to do with them before you write: remove them (the regex in the last line gave bellhere), replace them with a marker, or reject the row. Removing them silently changes the data, so I count them, log the count and write the rows that had any to a side file. The null bytes article has the CSV-side view of the same problem.

Elements, attributes and structure

Two conventions I use. The key of the row (id here) goes in an attribute, so a consumer can find a record by @id without descending, and the other columns are child elements, so text with newlines and markup is safe. And I put the root element and the row element names in as parameters. The big-file scripts below follow this shape: <customers><customer id="1"><first_name>Cy</first_name>.... An empty field becomes <phone_unparsed/>, which a parser reads as an empty string, so the difference between an empty field and a missing one is not representable. If it matters, omit the element for NULL and write an empty element for an empty string, or add xsi:nil, which needs a schema.

Writing 193,708 rows

ElementTree builds the whole document in memory, then writes it:

import csv, sys
import xml.etree.ElementTree as ET
src, out = sys.argv[1], sys.argv[2]
root = ET.Element("customers")
with open(src, newline="", encoding="utf-8") as f:
    for r in csv.DictReader(f):
        e = ET.SubElement(root, "customer", {"id": r["id"]})
        for k, v in r.items():
            if k != "id":
                ET.SubElement(e, k).text = v
ET.ElementTree(root).write(out, encoding="utf-8", xml_declaration=True)

/usr/bin/time -f "%e s, peak %M KB" python lab/conv/x_et.py work/val/customers.csv work/conv/c_et.xml

4.76 s, peak 375848 KB

It took 4.84 s and a peak of 375,584 KB, about 22 times the size of the CSV. A streaming writer, XMLGenerator from the standard library's SAX support, handles escaping and writes each element as it goes:

import csv, sys
from xml.sax.saxutils import XMLGenerator
src, out = sys.argv[1], sys.argv[2]
with open(src, newline="", encoding="utf-8") as f, open(out, "w", encoding="utf-8") as o:
    g = XMLGenerator(o, encoding="utf-8", short_empty_elements=True)
    g.startDocument()
    g.startElement("customers", {})
    for r in csv.DictReader(f):
        g.startElement("customer", {"id": r["id"]})
        for k, v in r.items():
            if k != "id":
                g.startElement(k, {}); g.characters(v); g.endElement(k)
        g.endElement("customer")
    g.endElement("customers")
    g.endDocument()

/usr/bin/time -f "%e s, peak %M KB" python lab/conv/x_sax.py work/val/customers.csv work/conv/c_sax.xml

3.22 s, peak 22584 KB

2.77 s and 22,336 KB. The memory no longer depends on the number of rows. XMLGenerator escapes text and attribute values, but it does not check names or control characters, so those two steps still apply. The output files were 63,310,759 and 62,997,177 bytes, about 3.6 times the CSV, because every value is wrapped in an opening and a closing tag; a first look at the head of the file:

ls -l work/conv/c_et.xml work/conv/c_sax.xml | awk '{printf "%-20s %d bytes\n", $9, $5}' | sed 's#work/conv/##'
head -c 420 work/conv/c_sax.xml; echo

c_et.xml   63310759 bytes
c_sax.xml  62997177 bytes
<?xml version="1.0" encoding="utf-8"?>
<customers><customer id="1"><first_name>Cy</first_name><last_name>Riga</last_name><email>[email protected]</email><phone>+14192381408</phone><phone_unparsed/><signup_date>2025-04-03</signup_date><balance>10485.58</balance><balance_currency>USD</balance_currency><units>4156</units><country>IT</country><status>active</status></customer><customer id="2"><first_name>Bo</first_name>

Reading it back

A consumer has the same choice. Parsing the file into a tree needs memory proportional to the file; iterparse processes elements as they close, and clear() frees each one:

import sys
import xml.etree.ElementTree as ET
n = 0
for event, el in ET.iterparse(sys.argv[1], events=("end",)):
    if el.tag == "customer":
        n += 1
        el.clear()
print(n, "customer elements read with iterparse")

/usr/bin/time -f "%e s, peak %M KB" python lab/conv/x_read.py work/conv/c_sax.xml
/usr/bin/time -f "%e s, peak %M KB" python -c "import xml.etree.ElementTree as ET; r = ET.parse('work/conv/c_et.xml').getroot(); print(len(r), 'customer elements parsed into a tree')"

193708 customer elements read with iterparse
1.77 s, peak 29640 KB
193708 customer elements parsed into a tree
1.90 s, peak 379692 KB

iterparse with clear() read all 193,708 elements in 1.32 s with a 29,640 KB peak. Parsing into a tree took 1.58 s and 379,768 KB. The element count matches the CSV record count, which is the first check to make on the result.

How to tell it worked

Parse the output with a real XML parser and compare the element count with the number of records in the CSV. Compare every value with the CSV text: whitespace in element text is preserved by the parser, but a validator or pretty-printer may change it. Put your awkward rows through it: ampersands, angle brackets, quotes, a newline, a control character and a header with a space. And decide on the control character policy before the data teaches you one.

When you need a schema or namespaces

A flat CSV maps to flat XML. If the consumer demands an XSD, namespaces, attributes in a defined order, or a particular element hierarchy, the converter is no longer a loop and the output has to be validated against the schema, which the standard library does not do. If the next step is to read the XML with a tool that expects types, XML has none either: <score>12.50</score> is text until the schema says otherwise. And for very large exports, ask whether the consumer could use JSON Lines or CSV directly. The conversion hub compares the targets.