Reading a SOAP Response and Pulling the Fields You Need

SOAP responses and pulling the payload out covered the envelope: header, body, fault. This article is about the next step, reading the fields in a real payload. The envelope is the easy part. The payload is where a response with a list, an optional field and a nil value turns a ten-line script into a source of wrong numbers, and the namespaces are a different kind of awkward than in a feed.

The response

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <soapenv:Header><ns2:Session xmlns:ns2="urn:session"><ns2:Id>S-77</ns2:Id></ns2:Session></soapenv:Header>
  <soapenv:Body>
    <ns1:GetOrdersResponse xmlns:ns1="urn:shop">
      <ns1:Page number="1" total="2"/>
      <Order id="1001"><Customer>Ana Lee</Customer><Total currency="USD">19.99</Total><Coupon xsi:nil="true"/></Order>
      <Order id="1002"><Customer>Bo Chen</Customer><Total currency="EUR">1250.00</Total><Coupon>SAVE5</Coupon></Order>
      <Order id="1003"><Customer/><Total currency="USD">0</Total></Order>
    </ns1:GetOrdersResponse>
  </soapenv:Body>
</soapenv:Envelope>

Look at the namespaces before you write a path. The envelope is soapenv. The header's session is in urn:session. The response wrapper GetOrdersResponse and the Page element are in urn:shop. And the Order elements are not prefixed: they have no namespace, although they are children of a namespaced element. This is not a mistake. A service whose schema leaves elementFormDefault at its default of unqualified produces exactly this: global elements (the operation and its response wrapper) are qualified by the target namespace, and local elements (the fields inside) are not. Plenty of real services do it.

The path that finds nothing

NS = {"s": "http://schemas.xmlsoap.org/soap/envelope/", "shop": "urn:shop", "xsi": "http://www.w3.org/2001/XMLSchema-instance"}
env = etree.parse("orders_resp.xml").getroot()
resp = env.find("s:Body/shop:GetOrdersResponse", NS)
print("shop:Order ->", len(resp.findall("shop:Order", NS)), "| plain Order ->", len(resp.findall("Order")))

children of response: ['Page{urn:shop}', 'Order{}', 'Order{}', 'Order{}']
shop:Order -> 0 | plain Order -> 3

The reflex when a namespace is involved is to prefix everything, and it fails here: shop:Order found 0, and the unprefixed Order found 3. The first line shows why, listing each child with its namespace in braces: Page is in urn:shop, the three Order elements in none. The method is to look at the actual tree before choosing the path, which a two-line print does. Then use the prefix on the elements that have a namespace and no prefix on those that do not, as the code does for Page and Order. Do not guess that "the response is in urn:shop, so its children are."

A tempting escape is local-name(): env.xpath("//*[local-name()='Order']") also returned 3. It works for any namespace layout, and it will also match an Order element in a different vocabulary, including inside an attached document in the header. Use it for exploration and for a quick check, and write the exact paths in code.

Reading a field: absent, empty, nil

The fields in a response can be missing in three ways, and SOAP adds one that plain XML formats rarely use: the explicit xsi:nil="true". A small reader names the states:

def val(el, name):
    c = el.find(name)
    if c is None: return "<absent>"
    if c.get("{%s}nil" % NS["xsi"]) == "true": return "<nil>"
    return c.text if c.text is not None else "<empty>"

1001 | customer: Ana Lee | coupon: <nil> | 19.99 USD
1002 | customer: Bo Chen | coupon: SAVE5 | 1250.00 EUR
1003 | customer: <empty> | coupon: <absent> | 0 USD

Three orders show all four outcomes. Order 1001 has a coupon that is explicitly nil: the service is saying "this field exists and has no value", which differs from not mentioning it. Order 1002 has a real code. Order 1003 has an empty Customer element and no Coupon at all. A plain findtext("Coupon") returns None for 1001 (no text) and 1003 (no element), and "" or the text otherwise, so the nil and the absent collapse. Whether the difference matters is a decision for the integration: a nil coupon may mean "cleared", an absent one "unchanged" in an update protocol. Write it down, and keep the reader so that the choice is visible. Where the service ships a WSDL, a generated client maps nil to None and absent to a missing attribute, with its own rules (calling a SOAP service from Python with zeep).

Numbers and currencies

Amounts arrive as text, with attributes that qualify them. Sum with Decimal, and group by the currency attribute:

{'USD': Decimal('19.99'), 'EUR': Decimal('1250.00')}

The zero order contributes nothing to the USD total. The two currencies are separate keys, since adding 19.99 dollars to 1,250.00 euros is meaningless. A total that arrives as 1,250.00 or 1250,00 from a service with a different locale will fail Decimal, and that is the right outcome: catch the error, log the order id and reject. Never convert with float or a locale-aware parser.

Lists and paging

The Page element says number="1" total="2": this is page 1 of 2. A client that reads the first response and stops silently loses half the orders, and nothing in the body announces the loss unless you read the paging fields. Loop until number == total, or until a response arrives with fewer items than a page, and check that the sum of the items across pages equals any count the service reports. Services vary in how they page (a continuation token, an offset, a date window), and the documentation, not the XML, tells you which; the XML tells you the numbers.

The header's session id, S-77, sits in a third namespace. It is returned from a path with that namespace ({urn:session}Id), and a client that keeps sessions has to send it back on the next request. Parse it as part of the response and do not treat the header as noise.

A defensive reading pattern

Four habits cover most of it.

1. Print the structure first, with namespaces, before writing paths.

2. Wrap field access in one reader that returns an explicit state, and decide once what each state means.

3. Convert at the edge, with exact types, and fail loudly on a value that does not parse.

4. Reconcile, counting the records in the response against the paging and against any total the service gives, and keep the raw envelope with the result.

The last one is for the day a number is questioned. The envelope is the evidence of what the service said.

Related reading

For the envelope and faults, see SOAP responses and pulling the payload out; for the generated-client route, calling a SOAP service from Python with zeep; for the cluster overview, working with XML feeds and APIs. Qualified and unqualified elements are explained in the XML Schema primer, and nil values in XML Schema Part 1. Test a path in the XPath tester.

When a typed client should read the response

You have a WSDL. Generate a client. It knows which elements are qualified, maps nil, converts types and validates the request, and it removes this whole class of mistakes.

The response is large. A tree parser and a hand-written reader are fine for hundreds of records. For millions, stream with iterparse and read one Order at a time.

The service changes. A typed client regenerated from the new WSDL shows the change as a compile error or a validation error. A hand-written reader shows it as a wrong number.

You only need to test it. For a one-off check or a monitor, the reader above is faster to write than a client.