Mixed Content in XML and Why It Hurts Parsing and Mapping

Open a recipe, an article or a legal clause in XML and you find text with markup in the middle of it: a paragraph with a bold phrase, a footnote reference, a link. This is called mixed content, and it is XML's reason for existing as a descendant of document markup languages. It is also the structure that breaks the pipelines built for data. If you have ever lost half a sentence in a conversion, this is why.

What the parser gives you

An element has mixed content when its children include both elements and non-whitespace text. The usual illustration is a sentence with inline markup:

<p>Call <b>Ana</b> before <i>noon</i>, please.</p>

A tree model with elements and attributes needs a place for each piece of text. The DOM puts text in separate text nodes. The ElementTree and lxml API do something more compact: an element has a text (the text before its first child) and every child has a tail (the text after the child, up to the next sibling or the parent's end):

from lxml import etree
p = etree.fromstring("<p>Call <b>Ana</b> before <i>noon</i>, please.</p>")
print("text:", repr(p.text))
for c in p:
    print(c.tag, repr(c.text), "tail:", repr(c.tail))
print("itertext:", "".join(p.itertext()))

text: 'Call '
b 'Ana' tail: ' before '
i 'noon' tail: ', please.'
itertext: Call Ana before noon, please.

The sentence is spread over four attributes in three objects. To get it back in order, you have to read p.text, then for each child its text and then its tail, recursively if children nest. itertext() does that for you and returns the plain text, which is the right call for search, display without styling, and counting words. It also discards which words were bold or italic.

The tail model surprises people the first time, because it makes text belong to the element before it rather than the one around it. Code that reads only .text of each child silently drops the connecting words. The bug is always the same: a field that looked fine in testing is truncated whenever real content has markup in it.

Why mapping fails

Data models are built on records with named fields. Mixed content is an ordered sequence of text runs and elements. Every mapping loses something:

To a string: you keep the reading text and lose the markup, so "Ana" is no longer bold. For search this is what you want. For display it is a loss.

To a string with tags kept: you keep everything and gain a string that needs a second parser. A database column with embedded HTML is a fine design when a browser renders it, and a bad one when the consumer wants fields.

To an object: you invent a list of nodes ([{"text": "Call "}, {"b": "Ana"}, ...]). This keeps order and markup and is a document model of your own, which no tool will understand.

To JSON with a generic converter: converters disagree. Some drop the text between elements, some collect it under a #text key and lose its position among the children. A paragraph with two separate text runs around one b comes out with the runs merged or one of them missing, depending on the tool. Check what yours does before trusting it with a document.

The shape of the problem is that data wants "an element is a container or a value", and mixed content says "an element is both".

Keeping data formats free of it

If you are designing a format for records, forbid mixed content. In a schema that means element-only content for containers and simple content for leaves, so each element is a list of elements or a text value and never both. XML Schema supports this directly: a complex type with mixed="false" (the default) rejects text between child elements, apart from whitespace.

When you do need rich text inside a record, give it its own leaf field and say what is in it. Put the paragraph in a <description format="html"> element as escaped text or CDATA, or as a nested document in its own namespace, and parse it separately when you need it. The record stays a record. The rich part is a value whose type is "HTML fragment", as it would be in a database column (entities, CDATA and escaping special characters covers the escaping rules).

Handling mixed content when you have it

When you receive a document format, process it as a document:

• Use XSLT for transformation. It was designed for mixed content: templates match elements and the built-in rules copy text through, so <xsl:apply-templates/> inside a paragraph template walks text and children in order (XSLT basics, templates and match).

• Use **itertext()** or string() in XPath to extract reading text: string(//p[1]) returns the same sentence as the Python call.

• Use a streaming parser for large corpora of documents, and handle start, end and text events in order instead of building a tree.

• When you must flatten, mark what you drop. Replace a footnote with [1], an emphasized word with its text, an image with its alt text, and keep a record of the rule so the output can be explained.

Do not strip whitespace from a document format. In a data file, indentation between elements is noise and remove_blank_text is safe. In mixed content the space between </b> and the next word is a real space (when whitespace in XML matters).

Related reading

The cluster hub is designing XML documents that age well. For the loading side, see dealing with mixed content and text in XML, and for the conversions that suffer from it, moving from XML to JSON gracefully. The definition is in the XML 1.0 recommendation on mixed content, and the tail attribute in the Python ElementTree documentation. Try an XPath string function in the XPath tester.

When mixed content is the right design

The thing you are encoding is prose. Articles, manuals, contracts, books and markup of any language-like text need inline structure. Flattening them is not a design, it is a loss. XHTML, DocBook, DITA and TEI exist for this.

People write it. Authors write paragraphs with emphasis and links; they will not fill a form with one field per phrase.

Order matters inside the text. Where a footnote falls or which word carries a link is data. A mixed content model keeps it, and a record model cannot.

The consumer is a renderer. If the next step is a browser or a typesetter, give it the markup. Do not convert to fields and back.