Loading Multiple Documents From One YAML File in Python

Kubernetes manifests, multi-resource Compose overrides, CI pipeline definitions and exported datasets often put several YAML documents in one file. The separator is a line with three dashes. If you load such a file with the function you use for single documents, it fails. If you load it with the right function, you get a generator you have to treat with care. This post covers the format and the Python calls, with the output.

The format

A YAML stream is a sequence of documents. A document starts at the beginning of the stream or after a line of three dashes (---), and it ends at the next ---, at a line of three dots (...) or at the end of the file:

---
a: 1
---
b: 2
---
- x

The first document is a mapping, the second another mapping, the third a list. The first --- is optional (a file without it is a single document). Each document is independent: it has its own root node, its own anchors and its own directives. A common real example is a Kubernetes file that has a Deployment document, a Service document and a ConfigMap document back to back.

Loading: safe_load fails, safe_load_all works

docs = "---\na: 1\n---\nb: 2\n---\n- x\n"
print(list(yaml.safe_load_all(docs)))
try:
    yaml.safe_load(docs)
except yaml.YAMLError as e:
    print("safe_load on 3 docs:", type(e).__name__, str(e).split("\n")[0][:80])

[{'a': 1}, {'b': 2}, ['x']]
safe_load on 3 docs: ComposerError expected a single document in the stream

safe_load_all returned the three documents: two dictionaries and a list. safe_load refused, with a ComposerError that says exactly what is wrong. Note that safe_load fails only when there is more than one document. A file with a leading --- and a single document loads fine, and a file whose second document is empty may or may not, depending on how you wrote the separator. Code that must accept either shape can use safe_load_all and take the first element, or check the length.

Another detail: a file of only comments or whitespace yields no documents from safe_load_all and None from safe_load. A file whose last line is --- yields a final document of None, which surprises code that iterates and indexes each document. Filter: docs = [d for d in yaml.safe_load_all(f) if d is not None].

The generator

safe_load_all returns a generator, and the work happens as you iterate. If the input is an open file, the file must still be open then:

with open("stack.yaml", encoding="utf-8") as f:
    docs = yaml.safe_load_all(f)       # nothing parsed yet
# file closed here: iterating docs now raises ValueError (I/O on closed file)

with open("stack.yaml", encoding="utf-8") as f:
    docs = list(yaml.safe_load_all(f))   # parsed while the file is open

The first version looks right and fails when you use docs. Either consume the generator inside the with block (for d in yaml.safe_load_all(f): ...) or turn it into a list. For very large streams the loop is better, since it holds one document at a time. A syntax error in the third document is raised while you are iterating to it, after you have already processed the first two. For all-or-nothing behavior, build the list first.

Writing several documents

print(yaml.safe_dump_all([{"a": 1}, {"b": 2}], explicit_start=True), end="")

---
a: 1
---
b: 2

safe_dump_all takes an iterable of objects and writes each as a document. With explicit_start=True every document, including the first, begins with ---, which is what most tools expect and what makes the output unambiguous. Without it, the first document has no marker. Add sort_keys=False and the other options from the single-document safe_dump (writing YAML from Python without ugly output).

Anchors stay inside a document

An anchor defined in one document is not visible in the next:

---
base: &b {x: 1}
use: *b
---
other: *b

The second document fails with an error about an undefined alias, because b was defined in the first. If several documents share definitions, repeat them, or load the shared part separately and merge in code. The same isolation protects you: a document cannot accidentally depend on a distant definition.

Working with them on the command line

yq handles streams: yq '.a' file.yaml runs the expression on every document, and yq eval-all '[.a]' -o=json -I=0 file.yaml collects across them. For a stream of two documents with a: 1 and a: 2 it printed 1, ---, 2 for the first form and [1,2] for the second. Use select(document_index == 1) to pick one. Splitting a stream into files is yq -s '.metadata.name' stack.yaml, which writes one file per document named by the expression (using yq to update many YAML files at once covers bulk workflows).

Patterns

Iterate and dispatch by kind. For a Kubernetes-style stream, loop over the documents and route each by its kind key. Validate each document against its own schema.

Keep order. The order of documents in the file is meaningful for some tools (a namespace must exist before the resources in it). safe_load_all preserves it, and so should your writer.

Count and report. When a stream is malformed, report which document (by index) failed, not only the line. A counter in the loop gives you that.

Prefer separate files for unrelated data. If the documents have nothing to do with each other, several files are easier to diff, review and validate.

Edge cases tested

Three small cases behave in ways that surprise people. A file ending with a lone separator line produced an extra document: loading a: 1 followed by --- gave [{'a': 1}, None], so a loop that indexes each document fails on the trailing None. A file containing only a comment gave an empty list from safe_load_all and None from safe_load, which are different empty results. And the stream-with-leading-separator --- and one mapping loaded through plain safe_load as {'a': 1}, so a leading marker alone does not make a file multi-document.

The closed-file case is the one that bites in practice: creating the generator inside a with block and using it after the block raised ValueError: I/O operation on closed file. Because the parse is lazy, the error appears at the point of use, far from the cause. Make the habit automatic: either loop inside the block or call list(...) there.

If you validate each document, give the error the index. Wrapping the loop in enumerate(yaml.safe_load_all(f), start=1) lets you report "document 3 of the stream is invalid", which saves a search through a long file.

Related reading

The cluster hub is reading and writing YAML in Python safely. The single-document call is explained in loading YAML in Python with PyYAML safe_load. Document markers are defined in section 9 of the YAML 1.2.2 specification, and PyYAML's functions in its documentation. Convert one document to JSON in the YAML to JSON tool.

When multi-document YAML is the wrong container

The consumer expects one document. Many tools read the first document only or reject a stream. Check before combining.

The documents are unrelated. A bundle of independent things in one file makes diffs, ownership and validation harder.

You need to merge them. If the intent is one configuration made of parts, use separate files and a merge step, or anchors within one document (splitting a large YAML config into several files).

The data is a long list of records. A list in one document is simpler than hundreds of documents, and parsers handle it better.