Using DynamoDB Maps and Lists as JSON Documents Safely

DynamoDB is a key-value store whose values happen to be documents. An item can hold nested maps and lists to a considerable depth, so a JSON payload fits naturally inside it, and the expression language can read and update the nested parts without reading the whole item first. It also has a set of rules about types and paths that differ from JSON and from every SQL database in this series. This article uses the Python boto3 library against DynamoDB Local to show how JSON maps to attribute types, how to update nested paths, and which errors to expect.

How JSON maps to attribute types

DynamoDB has its own type system, and the resource-level API in boto3 converts Python values to it. The mapping I observed for the item below:

from decimal import Decimal

item = {
    "id": "u1", "name": "Ada", "age": 36, "active": True, "nothing": None,
    "address": {"city": "Lyon", "geo": {"lat": Decimal("45.76"), "lon": Decimal("4.83")}},
    "tags": ["a", "b"],
    "orders": [{"sku": "A1", "qty": 2}, {"sku": "B2", "qty": 1}],
}
table.put_item(Item=item)

Reading the same item through the low-level client shows what is stored:

{"active": {"BOOL": true}, "address": {"M": {"city": {"S": "Lyon"}, "geo": {"M": {"lat": {"N": "45.76"}, "lon": {"N": "4.83"}}}}},
 "age": {"N": "36"}, "id": {"S": "u1"}, "name": {"S": "Ada"}, "nothing": {"NULL": true},
 "orders": {"L": [{"M": {"qty": {"N": "2"}, "sku": {"S": "A1"}}}, {"M": {"qty": {"N": "1"}, "sku": {"S": "B2"}}}]},
 "tags": {"L": [{"S": "a"}, {"S": "b"}]}}

So an object is an M (map), an array an L (list), a string S, a boolean BOOL, a null NULL, and a number N, whose value travels as a string. The string form is how DynamoDB keeps up to 38 digits of precision, and it is also why the Python side needs Decimal:

put_item with "price": 1.5  ->  TypeError: Float types are not supported. Use Decimal types instead.

The resource API raises on any float and wants Decimal("1.5"). When you read an item back, numbers arrive as Decimal (Decimal('36')), and if you serialize the item with json.dumps, it fails on them. Convert with a custom encoder (default=str turns them into strings, as I did to print the item, and a function converting to int or float is better for APIs). This is the same hazard as everywhere JSON numbers meet exact decimals, discussed in how JSON handles big numbers.

Other mapping notes: an empty string and an empty map or list are stored and read back as they went in ({"s": "", "l": [], "m": {}} came back unchanged), which older versions of the service did not allow for strings. A Python set becomes a string or number set (SS, NS), a native DynamoDB type with no JSON equivalent, so a JSON array should be a list and not a set unless you want set semantics (unique, unordered, all the same type). Binary values need bytes.

Reading nested paths

ProjectionExpression selects parts of an item, using dots for maps and brackets for lists:

table.get_item(Key={"id": "u1"},
               ProjectionExpression="address.city, orders[1].sku, tags[0]")["Item"]
# {'orders': [{'sku': 'B2'}], 'address': {'city': 'Lyon'}, 'tags': ['a']}

Look at orders. The projection of orders[1].sku gave a list with one object, [{'sku': 'B2'}], not the second element in place. The structure is preserved along the path, but list positions are compacted. If you read the result by index you will read the wrong element; take [0] or use the sku directly. A path that does not exist (address.zip, orders[5]) returned an empty item, with no error.

Updating nested paths

The UpdateExpression language works on paths, which is what makes DynamoDB documents workable: you change one value without sending the whole item.

# set a nested map value
UpdateExpression="SET address.city = :c"
# arithmetic on a list element
UpdateExpression="SET orders[0].qty = orders[0].qty + :n"
# append to a list
UpdateExpression="SET tags = list_append(tags, :t)"          # :t must itself be a list
# add a map only if it is not there
UpdateExpression="SET #p = if_not_exists(#p, :d)"
# remove list elements and map keys
UpdateExpression="REMOVE tags[0], address.geo.lon"

All of these worked. After SET orders[0].qty = orders[0].qty + 3 the first order's qty went from 2 to 5; list_append(tags, ["c"]) gave ['a','b','c']; and REMOVE tags[0], address.geo.lon left tags = ['b','c'] and a geo map with only lat. A few rules that tripped me:

• A missing parent is an error. SET address.zip.code = :c, where address.zip did not exist, raised ValidationException: The document path provided in the update expression is invalid for update. DynamoDB will not create intermediate maps. Create them first with SET address.zip = :m (with :m = {}) or set the whole parent at once. This is the opposite of SQLite's json_set, which creates parents (JSON functions in SQLite), and like PostgreSQL's, which leaves things unchanged.

• Reserved words need aliases. SET name = :n failed with Attribute name is a reserved keyword; reserved keyword: name, and so did a top-level attribute named missing. Use ExpressionAttributeNames={"#n": "name"} and SET #n = :n. The reserved list is long and includes common words (name, status, data, comment, count, size). Alias every attribute name in generated code.

• **REMOVE on a list shifts later elements.** REMOVE orders[0] left [{'sku': 'B2', ...}]: the second element became the first. If other code addresses by index, removals invalidate its indexes. With several removals in one expression, the indexes refer to the list as it was before the update: REMOVE l[0], l[2] on ['a','b','c','d'] left ['b','d']. Across separate updates the shift applies.

• Updates are atomic per item. All the clauses in one UpdateExpression apply together, and there is no multi-item atomicity except through transactions.

Conditions

ConditionExpression guards an update with a nested test: "address.city = :x" with :x = "Paris" made the update fail with ConditionalCheckFailedException, and with "Oslo" it succeeded. That is DynamoDB's compare-and-swap, and the way to avoid lost updates when two clients edit one item: read the item, compute the change, write it with a condition on the field you read (or on a version attribute you maintain). The rest of the cases follow the same logic as SQLite's where ... and old value pattern in storing settings as JSON in SQLite.

Finding documents by a nested field

DynamoDB indexes only top-level attributes. A nested field such as address.city cannot be a key or an index key. A scan with FilterExpression=Attr("address.city").eq("Oslo") returned the matching items (['u1', 'u2'] in my two-item table), and attribute_exists(address.geo) and a size test on a list work as filters. But a filter runs after DynamoDB reads the data, so a scan reads and bills every item in the table. For a lookup you need frequently, copy the value to a top-level attribute (city) and index it with a global secondary index. The document stays as the payload and the top-level attributes are the access paths.

Limits

Two limits shape the design. Item size is capped at 400 KB, including attribute names and nested content; put_item with a 400 KB string raised ValidationException: Item size has exceeded the maximum allowed size. Nesting depth is capped: DynamoDB Local rejected a document nested 40 levels deep (Nesting Levels have exceeded supported limits) and accepted 30. The service documents its own limit on nested levels (check the current figure for DynamoDB itself); stay well under it. Big documents belong in object storage with a pointer in the item, or split across items by a sort key.

Also remember that names count toward item size, so long key names in millions of items add up, and that reading one nested value still reads the whole item for the purpose of read capacity: a projection reduces the bytes returned, not the bytes billed.

When a DynamoDB document is not the right model

You need ad hoc queries across the document. Without secondary indexes on nested fields, flexible querying means scans. Use a relational or document engine with path indexes (working with JSONB in PostgreSQL).

Documents are large or grow without bound. Lists that append forever approach the item limit and make every write heavier.

You need to keep exact JSON text. DynamoDB normalizes numbers and does not preserve key order.

Concurrency on one document is high. Many writers to one item contend on it; shard the item or restructure so each writer owns a different item.

You are mapping an API payload straight in. Convert floats to Decimal, alias reserved names and decide what to do with the empty and null cases before the first production write.

The reference is the DynamoDB documentation on expression attribute names and update expressions and on service limits. The broader question of whether this fits is in choosing between a JSON column and a document database. To inspect a payload before storing it, use the JSON formatter.