Embedding Binary Data in XML With Base64 and When Not To

Sooner or later someone asks to put an image, a PDF or a signature inside an XML document. XML is a text format, and binary data is not text. The format has a fix, which is to encode the bytes as text, and the fix has a cost that is easy to underestimate. Whether to embed depends on how big the data is and who reads the file.

Why raw bytes cannot go in

An XML 1.0 document is a sequence of Unicode characters. Only a few control characters are allowed at all: tab, line feed and carriage return. The others, such as the null byte or \x01, are forbidden in the document, and that includes writing them as character references like � or . Parsers reject them with errors such as "invalid character" (invalid character in XML shows one). A CDATA section escapes markup characters like < and &, not control characters, so it does not help either. The bytes also may not be valid in the document's encoding: byte sequences in a binary file are rarely valid UTF-8.

So binary data has to be converted to text before it goes in, and back after it comes out.

Base64 and hex, measured

The two encodings people use are Base64, which turns every 3 bytes into 4 characters, and hexadecimal, which turns every byte into 2. Measure both on random bytes at three sizes:

import base64, os
for n in (1000, 100000, 1000000):
    raw = os.urandom(n)
    b = base64.b64encode(raw)
    h = raw.hex()
    print(n, len(b), round(len(b) / n, 3), len(h), round(len(h) / n, 3))

1000 1336 1.336 2000 2.0
100000 133336 1.333 200000 2.0
1000000 1333336 1.333 2000000 2.0

Base64 turns 1 MB into 1,333,336 characters, a third larger. Hex doubles it exactly. The small excess in Base64 at 1000 bytes is padding. In an XML document using UTF-8, each of these characters is one byte, so the file grows by the same factors. If the document is then compressed for transport, Base64 of data that was already compressed (JPEG, ZIP, PDF) is mostly incompressible, and you pay the full third on the wire.

Choose Base64 unless a standard requires hex. It is smaller and universally supported. Some formats require hex for short values such as hashes (xs:hexBinary describes it), and that is fine at 32 bytes.

Declaring it

XML Schema has a type for each: xs:base64Binary and xs:hexBinary. A schema that uses them lets a validator check that the text contains only legal characters and the right padding, and lets a code generator produce a byte[] rather than a string:

<xs:element name="signature" type="xs:base64Binary"/>

<signature>iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR4nGP4z8DwHwAFAAH/q842iQAAAABJRU5ErkJggg==</signature>

Add attributes for what the bytes are: a media type and, if useful, a file name and a length. The decoder does not need them, and the human or the next system often does:

<attachment type="image/png" name="logo.png">iVBORw0K...</attachment>

Line wrapping is a trap. Some producers wrap Base64 every 76 characters, as MIME does, and the line breaks become whitespace in the text node. xs:base64Binary allows whitespace in the lexical form, and most decoders ignore it, but a strict one may not. If you are the producer, do not wrap. If you are the consumer, strip whitespace before decoding.

The memory cost

The disk size is only part of it. A DOM parser loads the whole document, so the encoded text is in memory as a string, and decoding makes a second copy as bytes. A 100 MB attachment encodes to about 133 MB of text, and the parser needs that and more, since text nodes carry their own overhead. A streaming parser can handle it in pieces, but many libraries deliver a text node as one string anyway. If documents can carry large attachments, set a limit on size and depth at the edge (limiting XML size and depth for uploads).

Alternatives for large data

A reference. Put a URL, a path or an id in the XML and send the bytes by another route. The document stays small and the file can be streamed, cached and scanned for malware on its own. The cost is two things to keep consistent and a link that can break.

A package. Zip the XML and the files together with a manifest, which is how Office documents work (reading Office Open XML (docx) shows the layout). The XML stays plain and the binaries are separate entries.

MTOM or MIME multipart. SOAP's Message Transmission Optimization Mechanism sends the binary as its own MIME part and leaves an xop:Include reference in the XML. It avoids the Base64 overhead and is supported by the major SOAP stacks, with the downside that it needs a transport that can carry multipart bodies.

Hash plus reference. For integrity-critical data, include the SHA-256 of the file in the XML next to the reference. The consumer fetches the file and checks.

Checking what you received

If you consume documents with embedded binary data, validate it before you trust it. Decode with a strict decoder so that stray characters raise an error: in Python, base64.b64decode(text, validate=True) rejects anything outside the alphabet, which catches a text node that someone truncated or edited. Compare the decoded length with a declared length if the document carries one, and compare a hash if it carries a hash. Check the leading bytes against the declared media type, since a PNG starts with the eight bytes 89 50 4E 47 0D 0A 1A 0A and a PDF with %PDF. A file that claims to be an image and decodes to something else is a reason to reject the document, not to guess.

Related reading

The cluster hub is designing XML documents that age well. For the broader choice of format when binary is involved, see XML versus Protocol Buffers for services. Encoding is covered in XML encoding declarations and wrong encodings. The types are defined in XML Schema Part 2, and MTOM in the SOAP MTOM recommendation. Format a document with embedded data in the XML formatter.

When binary data should stay out of the XML

The payload is large. Past a few hundred kilobytes per attachment, a reference or a package is better in every respect except the single-file convenience.

The XML is inspected by people or diffed. A 40 KB line of Base64 destroys a diff and an editor. Keep it out of anything reviewed.

The same binary repeats across documents. Store it once and refer to it by id or hash. Embedding it in every document multiplies storage and transfer.

You need to scan or transform the file. Antivirus, image resizing and OCR all work on the file, not on a string inside a document. Decode early and handle files as files.