SOAP Responses and Pulling the Payload Out With XPath

SOAP is the envelope format that big enterprise systems, banks, airlines and government services used for years and still use. You may be handed a WSDL and a URL, or just a captured response in a ticket and a request to find out what the number in it is. You do not need a SOAP library to read a response. The envelope is XML, the structure is fixed, and a few lines pull out the payload.

The envelope

<?xml version="1.0" encoding="UTF-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
  <soap:Header><h:Trace xmlns:h="urn:trace">abc</h:Trace></soap:Header>
  <soap:Body>
    <m:GetPriceResponse xmlns:m="urn:shop">
      <m:Price currency="USD">19.99</m:Price>
      <m:InStock>true</m:InStock>
    </m:GetPriceResponse>
  </soap:Body>
</soap:Envelope>

SOAP 1.1 has a fixed skeleton. The root is Envelope in the namespace http://schemas.xmlsoap.org/soap/envelope/. An optional Header holds metadata for intermediaries and the receiver: security tokens, trace ids, routing. A required Body holds the message. In a response the Body has exactly one child, the operation's response element, here GetPriceResponse, which belongs to the service's namespace (urn:shop), not the envelope's.

So the document has three namespaces: the envelope, the header's own (urn:trace) and the service's. The prefixes (soap, h, m) are the sender's choice. Other servers use s, SOAP-ENV, ns2, and the numbering of generated prefixes changes between versions. Your code must not depend on them.

Taking the payload out

from lxml import etree

SO = {"soap": "http://schemas.xmlsoap.org/soap/envelope/", "m": "urn:shop"}
env = etree.parse("soap_ok.xml").getroot()
body = env.find("soap:Body", SO)
resp = body[0]
print("payload:", etree.QName(resp).localname, "| price", resp.findtext("m:Price", namespaces=SO),
      resp.find("m:Price", SO).get("currency"), "| header:", [etree.QName(h).localname for h in env.find("soap:Header", SO)])
print("naive Body/*:", len(etree.parse("soap_ok.xml").getroot().findall("Body/*")))

payload: GetPriceResponse | price 19.99 USD | header: ['Trace']
naive Body/*: 0

The payload element is body[0], the first (and only) child of the Body, and its local name is GetPriceResponse. The price and its currency attribute are read with the service's namespace, and the header's one child is a Trace. The last line is the usual failure: an unprefixed Body/* returns zero elements, because Body is in the envelope namespace. The fix is the prefix map, and since the service's namespace is not known in advance for a generic reader, take the payload by position (body[0]) and read its children by local name when you do not care about the namespace.

Do not search with //Price or //*[local-name()='Price'] without thinking. The first never matches and the second matches any element named Price in any namespace, including one nested in a header. Anchor to the Body.

For a number, parse text with Decimal, and read the currency from the attribute that came with it. A price of 19.99 with no currency is only half of a fact.

Faults

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
  <soap:Body>
    <soap:Fault><faultcode>soap:Client</faultcode><faultstring>Unknown SKU</faultstring><detail><e:Code xmlns:e="urn:err">404</e:Code></detail></soap:Fault>
  </soap:Body>
</soap:Envelope>

fault = body.find("soap:Fault", SO)
if fault is not None:
    print("FAULT", fault.findtext("faultcode"), "|", fault.findtext("faultstring"), "| detail:", fault.xpath("string(detail//*)"))

soap_fault.xml FAULT soap:Client | Unknown SKU | detail: 404

A fault replaces the response element with soap:Fault. In SOAP 1.1 it has faultcode, faultstring, optionally faultactor and detail. Note that faultcode, faultstring and detail are not namespaced: they are children of a namespaced element but have no namespace themselves, which is why findtext("faultcode") works without a prefix. That is how the 1.1 schema is written, and it is another reason not to assume. The faultcode value is a qualified name (soap:Client, soap:Server, soap:MustUnderstand, soap:VersionMismatch): Client means the request was wrong, Server means the service failed. The detail holds application-specific content, here an error code in its own namespace.

A server normally sends a fault with HTTP status 500, and the body is still well-formed XML. A client that checks the status and raises before it reads the body loses the explanation. Read the body when the status is 500 and the content type is XML.

Keeping the response for later

When a call matters (a payment, a booking), store the raw response bytes next to whatever you extracted, with the time and the request id. The extracted price is an interpretation; the envelope is the evidence. A later dispute is settled by the bytes, and a later bug in your parser is repaired by reparsing them. Store them as received, not re-serialized, because serialization changes whitespace, namespace prefixes and attribute order, and a signature in the header, if there is one, covers the exact form.

SOAP 1.2

SOAP 1.2 uses a different namespace (http://www.w3.org/2003/05/soap-envelope) and a different fault structure: Fault with Code/Value, Reason/Text and Detail, with the capitalized names and namespaced children. The content type changes to application/soap+xml. A reader that must accept both checks the envelope's namespace first and picks the right map. The two envelopes differ enough that a single path expression for faults does not work across them.

Reading without the library, and when to use one

For a captured response, a monitoring check or a one-off integration, the script above is enough, and it is easy to test. For a full integration, a client generated from the WSDL (zeep in Python, wsimport in Java, the .NET service reference) handles the request side, type conversion, headers such as WS-Security, attachments (MTOM) and the many faults and edge cases. See calling a SOAP service from Python with zeep for the library route and reading a SOAP response and pulling the fields you need for the field-extraction side of the same task.

Safety

SOAP responses come from a network. Parse them with entity expansion and network access disabled, as for any XML you did not write (disabling DTDs and external entities in Java and Python). SOAP forbids DTDs in messages, so rejecting any document with a DOCTYPE is legitimate here and closes a whole class of attacks. Log the envelope when a call fails, but remove header tokens and any personal data first.

Related reading

For the broader map, see common XML formats you will meet in real work and working with XML feeds and APIs. The specifications are SOAP 1.1 and SOAP 1.2 Part 1. Test the path expressions in the XPath tester.

When a SOAP client library is the better way

You are sending requests. Building envelopes by hand, with the right headers and types, is where mistakes concentrate. Generate a client from the WSDL.

The service uses WS-Security, MTOM or WS-Addressing. These add signatures, attachments and routing headers. A library implements them; a script does not.

You need types. A generated client gives you objects and converts numbers and dates. The raw reader gives you strings.

The service is long-lived and critical. Contract tests against the WSDL belong in code that the team maintains.