Validating XML Against an XSD Schema Step by Step Guide

A feed arrives, and the first question is whether it is what you were promised. If you read it before asking, the answer comes later and at a higher price: a failed load at row 90,000, a column of nulls, a report with a wrong total. Validation moves that question to the door. An XML schema is a contract that a program can check, and an XSD is the form of it that almost every partner and every toolchain understands.

This guide covers the whole practice: what validation does and does not guarantee, how to write a schema that is worth having, how to run it from the command line and from Python, how to read the errors, what to do about the rules a schema cannot hold, how to handle large files and how to change a schema without breaking the people who depend on it. Each spoke in this cluster goes deeper on one part, and they are linked where they apply.

What validation guarantees

Two different checks hide behind the word, and the order matters.

Well-formedness is the syntax of XML: tags nest and close, attributes are quoted, special characters are escaped, there is one root. A parser checks it, always, and a file that fails is not XML at all (well-formed versus valid XML). Validity is conformance to a grammar, the schema. A file can be well-formed and invalid (the tags are right but there is an element nobody declared). It cannot be valid and not well-formed.

xmllint makes the order visible through its exit code. On the test files:

good.xml        exit 0   validates
e_enum.xml      exit 3   schema violation
e_wf.xml        exit 1   parser error, schema never consulted

Exit 0 is success, 1 is a syntax error (the parser stopped), 3 is a schema violation, and 5 means the schema itself did not compile. The distinction pays off in operations: exit 1 is a conversation with whoever produced a broken file, and exit 3 is a conversation about the contract.

Validation guarantees that the document has the structure and value formats the schema describes. It does not guarantee that the content is true. A schema-valid order can have a total that disagrees with its lines, a customer that does not exist and a date in the year 2090. And it does not tell you the document is safe: a valid document can still be a very large one, or contain entities that an unsafe parser would expand. Those are separate controls (see the security cluster, starting with XML security problems to close before production).

Writing a schema worth having

A schema earns its keep by rejecting things. One that accepts anything is a very long way of saying <xs:any/>. The details are in writing a basic XSD schema, and the short version is that you use six constructs for most of the work.

With those, a forty-line schema for an orders file rejected twelve of thirteen test files, each for a different reason. The ones I would add to any schema are the named simple types for anything with a shape (SKUs, status codes, identifiers), because they give the error messages a name to point to and keep a rule in one place.

Two cautions. Do not make the schema stricter than the business, since every rejection costs a human a conversation. A field that is "usually" six digits is not a six-digit pattern. And do not leave it so loose that it checks nothing: a status declared xs:string accepts PAID, paid , Paid and pending-review, and the load that follows has to cope with all of them.

If you have no schema and a pile of samples, an inference tool gives you a draft. The one I ran, .NET's XmlSchemaInference, got the structure right (optional, repeated, required) and the types wrong: it typed ids and quantities as xs:unsignedByte, a 0 to 255 range chosen because the sample values were small, and it could not infer enumerations or patterns from two samples. Treat the output as a first draft and see generating an XSD from sample XML for the review list.

Running it

From the command line:

xmllint --noout --nonet --schema shop.xsd incoming.xml

That is the whole gate. --nonet stops the validator from fetching URLs referenced by the document or schema, and should be on whenever the input is not yours. The details, the message format and the stream mode are in validating XML against XSD with xmllint.

From Python, the schema is compiled once and called on each document:

from lxml import etree
schema = etree.XMLSchema(etree.parse("shop.xsd"))
if not schema.validate(etree.parse("incoming.xml")):
    for err in schema.error_log:
        print(err.line, err.type_name, err.path, err.message)

The difference from the command line is the error log: one call returned four independent errors from a single bad document, each with a line, a symbolic type and a path. A syntax error arrives as an exception at parse time instead, so a loader has two handlers. The full pattern, with assertValid and the parser option that validates while parsing, is in validating XML in Python with lxml and XSD.

Reading the errors

A message is made of the element, the rule and what was found. Twelve distinct rule types appeared across the thirteen test files, here with the path lxml attached to each:

file           result         first problem
e_anyattr.xml  invalid        SCHEMAV_CVC_COMPLEX_TYPE_3_2_1 at /orders/order/item
e_attr.xml     invalid        SCHEMAV_CVC_COMPLEX_TYPE_4 at /orders/order
e_dec.xml      invalid        SCHEMAV_CVC_DATATYPE_VALID_1_2_1 at /orders/order/item
e_enum.xml     invalid        SCHEMAV_CVC_ENUMERATION_VALID at /orders/order/status
e_extra.xml    invalid        SCHEMAV_ELEMENT_CONTENT at /orders/order/coupon
e_int.xml      invalid        SCHEMAV_CVC_DATATYPE_VALID_1_2_1 at /orders/order
e_missing.xml  invalid        SCHEMAV_ELEMENT_CONTENT at /orders/order
e_ns.xml       invalid        SCHEMAV_CVC_ELT_1 at /*
e_order.xml    invalid        SCHEMAV_ELEMENT_CONTENT at /orders/order/status
e_pat.xml      invalid        SCHEMAV_CVC_PATTERN_VALID at /orders/order/item
e_qty.xml      invalid        SCHEMAV_CVC_DATATYPE_VALID_1_2_1 at /orders/order/item
e_root.xml     invalid        SCHEMAV_CVC_ELT_1 at /order
e_wf.xml       not well-formed  line 4: Opening and ending tag mismatch: order line 3 and
good.xml       valid
{'invalid': 12, 'not well-formed': 1, 'valid': 1}

That table is a ten-line script around the error log, and it is the report I would send to a partner: the file, the verdict, the first rule broken and where. Grouping by the type name rather than by the message means that a hundred files with the same defect are one line, not a hundred. The type names are stable across a libxml2 upgrade and the wording is not.

The messages themselves are covered one by one in common XSD validation errors explained. Three patterns are worth knowing at a glance, because they account for most real incidents:

• "This element is not expected. Expected is ( x )" is either a wrong order or an undeclared element. Look the element up in the schema to tell which.

• "No matching global declaration available for the validation root" is nearly always a namespace mismatch between the document and the schema's targetNamespace, or a document of a different type entirely.

• "is not a valid value of the atomic type" is a value in the wrong form: a comma for a decimal point, a date in the local format, a thousands separator.

Rules a schema cannot hold

Run the same order through XSD, RELAX NG and Schematron, and the differences come out. An order whose total did not equal quantity times price passed both XSD and RELAX NG and failed Schematron. A negative price passed all structure checks too, because xs:decimal allows it. Reordering the children failed the XSD (a sequence) and passed RELAX NG (an interleave). The comparison and the setup are in choosing between XSD, RELAX NG and Schematron.

The practical stack that has served me is layered. The first layer is well-formedness. The second is XSD, usually the partner's own schema, for shape and types. The third is Schematron or code for relationships between fields, uniqueness, and anything that needs a lookup. The fourth is the database's constraints, which are the only thing that holds for data that arrives another way. Each layer reports separately, because "3 files failed shape, 2 failed business rules" tells different people to act.

A caution about Schematron through lxml: it compiles to XSLT 1.0, so abs, min, max and regular expressions are not available. My first assertion used abs and raised Unregistered function; squaring the difference was the workaround.

Large files and memory

The cost to watch is memory, not time. On the 36.6 MB, 200,000-order file:

The fourth row is the trap. Passing the schema to iterparse validates while streaming, and it holds the whole tree anyway, unless you also select the record and clear it. The fifth row is the fix, the same idiom as in loading large XML files into a database in chunks. Validating and loading in one pass is possible, with the choice that a bad record then stops the load where it stands. If you would rather quarantine bad records, validate the whole file first with xmllint --stream and load afterwards.

Validation is rarely the slow part. A gigabyte of XML validates in a few seconds to a few tens of seconds, and the parse is what takes the time. If a nightly job is slow, look at the load and the database first.

Changing a schema safely

Schemas change, and the people who read your feed do not all change with them. A change is safe in one direction or the other, and rarely both. On six proposed changes, tested with saved old documents against the new schema and new-style documents against the old one:

The two changes people call safe (an optional element and a new enumeration value) break every consumer still validating with the old schema as soon as the producer uses them. The answer is an order of deployment (consumers first), an extension point in the schema from the start (xs:any in another namespace), and a version in the namespace for the changes that break both ways. The script and its reasoning are in testing feed compatibility before a schema change. Save the feeds you receive: a corpus of real documents is what makes the test meaningful.

Namespaces, imports and remote resources

A schema with a targetNamespace validates only documents in that namespace, and that is where "No matching global declaration" comes from. Be deliberate about it. If the sender uses a namespace, your schema has to declare it, and your XPath queries have to use it (how XML namespaces work). elementFormDefault="qualified" means local elements are in the namespace too, which is what most real schemas use, and the setting is worth fixing at the top of a schema even when you have no namespace yet.

Schemas can import and include other files, and documents can carry an xsi:schemaLocation hint. Two rules keep this from becoming a problem. Keep the full set of schema files in your repository and refer to them by relative path, so validation does not depend on a remote server. And disable network access for anything you did not write. A validator that fetches whatever URL a document names is a way for a sender to make your server request addresses of their choosing, and it can stall your pipeline when a remote host is down. xmllint --nonet does it; lxml's parser has no_network=True by default.

Whitespace, nil and extension points

Three schema features decide whether a contract is livable. I tested each on the orders schema.

Whitespace in values. A pattern is compared against the value exactly as it arrives. A SKU written A-100 with a trailing space fails a pattern on xs:string, and passes when the type derives from xs:token, which collapses whitespace before the pattern is applied:

trailing space in sku, xs:string base            False | Element 'item', attribute 'sku': [facet 'pattern'] The value 'A-100 ' is not accepted by t
trailing space in sku, xs:token base             True

Which you want depends on the contract. A strict schema on xs:string is honest about the space and annoying to a sender whose export pads every field. xs:token is forgiving at the door, and then your loader still has to strip the value, because the schema does not change the text you read afterwards. The failure to avoid is the invisible one: a message that says the value A-100 is not accepted by the pattern, where the reader cannot see the space.

Explicit nulls. A sender that wants to say "this element is present and has no value" can use xsi:nil="true", and the schema has to allow it with nillable="true" on the element. Without it:

xsi:nil note, not nillable                       False | Element 'note': The element is not 'nillable'.
xsi:nil note, nillable                           True

This is the one standard way XML has to distinguish a missing value from an empty string, and few feeds use it. If a partner does, your loader has to map it deliberately (handling optional XML elements and missing values).

Extension points. A schema that lists every element rejects the first addition. A wildcard says where additions are welcome. With xs:any namespace="##other" processContents="lax" placed after the item list, a foreign-namespace element passes where it failed before:

foreign element after item, no wildcard          False | Element '{urn:ext}ext': This element is not expected. Expected is ( item ).
foreign element after item, xs:any lax           True
unqualified element after item, xs:any other     False | Element 'ext': This element is not expected. Expected is one of ( item, ##other* ).

The third line is the catch. ##other means "any namespace other than the schema's own", and an element with no namespace is not accepted, so a sender who adds a plain <ext> element is still rejected. Choose the namespace setting deliberately (##any, ##other or ##targetNamespace) and tell senders what extensions look like. Extension points are what make the first two rows of the compatibility table survivable: if every element in the old schema had one, an added optional element in another namespace would not have failed the old validators.

A worked session

This is the order of work I follow when a new feed arrives with no schema, from first file to a gate that runs unattended.

1. Check the syntax. xmllint --noout file.xml. If it fails, you are done for now, and the message goes back to the sender.

2. Profile the structure. Count the element and attribute paths (counting elements and attributes in an unknown XML file). The profile tells you the record, what repeats, what is rare.

3. Infer a draft. Run an inference tool on as many samples as you have (generating an XSD from sample XML). Do not trust the types.

4. Edit the draft into a contract. Replace guessed numeric types, add enumerations from the distinct values you counted, add patterns for identifiers, name the types, decide on order and extension points.

5. Test with every sample. All of them must validate. Then write the bad ones: one per rule, each named for what it breaks (e_enum.xml, e_dec.xml). Run the set and compare with the expected exit codes.

6. Wrap it in the gate. The shell fragment or the Python function, with --nonet, a size limit and a report.

7. Keep the samples. The good and the bad ones live in the repository next to the schema and run in CI.

8. Monitor. Count verdicts by day. An increase in rejects is information.

The first six steps took an afternoon for the orders feed. Step 8 is what pays for them months later.

Mistakes I keep seeing

• Validating before checking the encoding. A document in an unexpected encoding fails as "not well-formed" with an error about an invalid byte. That is a parser problem, and it needs the techniques in encoding mismatch errors in XML documents, not a schema change.

• A schema generated from one file. It accepts that file and few others. Add samples, widen the types.

• Patterns copied from a language with anchors. XSD patterns are matched against the whole value, so anchors are not needed, and ^ and $ are ordinary characters there rather than anchors. The dialect also lacks lookahead and backreferences.

• Treating a default as a guarantee. default="1" on an attribute documents what an absent attribute means, and a validator that only validates does not insert it. Code the default in the loader.

• Fixing the schema to make a failing file pass. Sometimes the schema is wrong. Often the file is. Look at the file before you edit the schema, and ask the sender when the answer is not obvious.

• Ignoring warnings at schema load. A schema that compiles with a warning about an ambiguous content model (the unique particle attribution rule) is already telling you that two branches of a choice can start with the same element. Fix it, because other validators may reject it outright.

• Not versioning. An unversioned schema edited in place is a breaking change without a notice.

Schema review checklist

When I review someone else's XSD, or my own, I go down this list. It takes ten minutes and finds most of the problems.

• Every identifier, code and status has a named simple type with a pattern or enumeration.

• Numeric types are as wide as the data can become, not as the samples were.

• Decimal fields have fractionDigits where the money or measurement requires it.

• Dates and timestamps use xs:date and xs:dateTime, and the time zone rule is written down.

• Optional versus required matches the business, and was checked against a month of real documents.

• The order of children is a sequence only where order is part of the contract.

• There is an extension point, and its namespace setting is deliberate.

• elementFormDefault and the targetNamespace are set on purpose.

• There is at least one passing and one failing test file per rule.

• The schema has a version, and the version appears in the namespace or the root element.

• A comment says where the schema came from and who to ask.

Fitting it into a pipeline

A pipeline that takes feeds from outside has a few stations, and validation is the second:

1. Receive and store the original, unmodified, with a timestamp and a checksum.

2. Check well-formedness and the schema. Reject with a report on failure. Keep the rejected file.

3. Check business rules (Schematron or code). Quarantine individual records if the format allows.

4. Load to staging, then to the real tables, in batches (turning XML feeds into relational tables).

5. Reconcile: counts and one sum, source against destination.

Make each station emit a count and a reason. "1,204 files received, 1,190 valid, 11 failed shape (7 for the same missing attribute), 3 failed business rules" is a monitoring line you can alert on. A sudden jump in rejects almost always means the sender changed something, and you want to know on the day.

Write the report for a reader who will not open the schema. Name the file, the rule in plain words, the element path and the value that was found, and put the schema version in the header so a reply can say which contract was applied. Send the same report to your own team, because the reject counts per sender are the data you need when someone asks whether a partner is getting better or worse.

Add regression tests for the schema itself: a folder of documents that must validate and a folder that must fail, with the expected rule for each. The twelve bad files above are a starting point. When the schema changes, those tests catch the case where a bad file silently starts validating.

What validation is for

Validation is also the cheapest place to build trust with a partner. A report that names the rule, the path and the value ends a dispute in one message, and a schema that both sides can run turns "your file is wrong" into a failing test they can reproduce on their own machine. That is worth more than the rejection itself.

It is worth being clear about the purpose. Validation is a cheap, early, precise rejection of bad input. It is also a form of documentation: the schema says what the format is, in a language machines can check, and the examples that fail say what it is not. It is not a security boundary on its own and not a substitute for checking the data against reality. And it is a conversation tool: the report you send the sender, with a rule and a path, ends arguments about whether the file was wrong.

Where to read next

The spokes of this cluster: writing a basic XSD schema, validating XML against XSD with xmllint, validating XML in Python with lxml and XSD, common XSD validation errors explained, choosing between XSD, RELAX NG and Schematron, generating an XSD from sample XML and testing feed compatibility before a schema change. Related: dealing with XML feeds that break their own schema, fixing XML parse errors fast for the syntax gate and XML DTDs and when they still matter for the older schema language.

The primary sources are the XML Schema Part 0 primer, the Structures and Datatypes specifications, the libxml2 documentation for xmllint, and the lxml validation page. The XPath tester is useful for developing the paths in Schematron assertions.

When validation is not the right control

You need to trust the sender's data, not its shape. Validity says the document looks right. Reconcile totals, check keys against reference data and compare against the previous batch.

The input is hostile. A schema check happens after parsing, and parsing is where entity expansion and external entity attacks live. Configure the parser first (disable DTDs and external entities, limit size and depth), then validate (stopping the billion laughs attack in your parser).

The format changes every week. A schema nobody maintains is worse than none, because it rejects good files and gets disabled. For a volatile feed, validate the small core you rely on, and monitor the rest with counts.

You own both ends of a private feed. If the producer and the consumer are the same team, a test that round-trips the data may do more than a schema. Keep the schema anyway for the day a third party appears.

The files are tiny and rare. A hand-edited config read by one program is better served by the program's own error messages.