Splitting a Large YAML Config Into Several Files Cleanly

A configuration file that grew to a thousand lines is hard to read, hard to review and a magnet for merge conflicts. The obvious fix is to split it, and YAML offers no include statement for that. Teams invent conventions, and two of them cover most cases: fragments that are merged in a defined order, and a custom tag that includes another file. This post shows both working, with the output, and says when each is a good idea and when splitting is the wrong move.

Option 1: fragments merged in order

The convention comes from Unix conf.d directories: a folder of small files, applied in sorted order, so the numeric prefix sets the precedence. Three fragments:

server: {host: localhost, port: 8080}
features: [search]
logging: {level: info}

server: {host: app.example, tls: true}
logging: {level: warn}

logging: {level: debug}

A loader reads them in sorted order and folds them with a deep merge. After each file:

after 10-base.yaml  -> {'server': {'host': 'localhost', 'port': 8080}, 'features': ['search'], 'logging': {'level': 'info'}}
after 20-prod.yaml  -> {'server': {'host': 'app.example', 'port': 8080, 'tls': True}, 'features': ['search'], 'logging': {'level': 'warn'}}
after 30-local.yaml -> {'server': {'host': 'app.example', 'port': 8080, 'tls': True}, 'features': ['search'], 'logging': {'level': 'debug'}}

The effective configuration has the production host, the base port, TLS from production and debug logging from the local file. Each file contributes only what it needs to change. The code is the merge function and a loop over sorted(glob.glob("conf.d/*.yaml")):

cfg = {}
for f in sorted(glob.glob("conf.d/*.yaml")):
    cfg = merge(cfg, yaml.safe_load(open(f)))

Advantages: every fragment is plain YAML that any tool can read and lint, there is no custom syntax, and the order is visible in the file names. Weaknesses: the merge semantics must be specified. A deep merge combines mappings and replaces scalars and lists, so a fragment that sets features: [billing] replaces the base list instead of adding to it. If you want accumulation, the merge needs an append policy for lists, or the fragments need to use a different structure (a mapping keyed by name, which merges naturally) (merging two YAML files in Python with a deep merge covers the policy choices). A null in a later fragment overwrites an earlier value, which may or may not be what the author meant.

Because the merge happens before the application reads anything, you can run it once at build or deploy time and ship the merged file. That keeps the runtime simple: the application loads one file.

Option 2: an include tag

The alternative is for one file to refer to another, with a custom tag:

app: !include conf.d/10-base.yaml
extra: !include conf.d/30-local.yaml

!include is not a standard tag, so the loader must know it. A subclass of SafeLoader with a registered constructor does it:

class L(yaml.SafeLoader): pass

def include(loader, node):
    path = os.path.join(os.path.dirname(main_path), loader.construct_scalar(node))
    return yaml.load(open(path), Loader=L)

L.add_constructor("!include", include)
cfg = yaml.load(open("main.yaml"), Loader=L)

!include: {'app': {'server': {'host': 'localhost', 'port': 8080}, 'features': ['search'], 'logging': {'level': 'info'}}, 'extra': {'logging': {'level': 'debug'}}}
safe_load on !include: ConstructorError could not determine a constructor for the tag '!include'

The included files appear as values under the keys that reference them: app holds the base fragment, extra the local one. The same document through plain safe_load fails with ConstructorError: could not determine a constructor for the tag '!include', which is the loader doing its job. That is also the weakness: every program that reads the file must use your loader. The linter does not know the tag, yq does not know it, the editor's schema validation does not know it, and a colleague who loads the file with safe_load gets an error. The structure also changes: the data is nested under the including key, so the include is a position in the tree and not a merge.

Resolve paths relative to the including file, guard against cycles (a file that includes itself) and never allow an include path to escape the config directory when the files are untrusted. An include tag is code that opens files, and it should be written as carefully as any other.

Which to choose

For layered configuration, base plus environment plus local, fragments are the better fit. For a configuration made of unrelated sections (one file per subsystem), an include tag can read well, but it is just as easy to have the application load several files by name and keep each section separate. If you can avoid a custom tag, do.

Some tools have their own mechanisms: Docker Compose has an include element and extends, Kubernetes uses Kustomize overlays, and CI systems have include keywords. When your file belongs to such a tool, use its mechanism, because it is validated and understood by the tool's schema and by editors that know the tool.

Making a split config debuggable

Splitting hides the answer to "why is this value 8080?". Add a way to print the merged result and where each value came from. Track the file of origin during the merge and print a table:

/server/host       = 'app.example' from 20-prod.yaml
/server/port       = 8080          from 10-base.yaml
/server/tls        = True          from 20-prod.yaml
/features          = ['search']    from 10-base.yaml
/logging/level     = 'debug'       from 30-local.yaml

A table like this settles arguments about precedence in seconds. Put it behind a flag such as --show-config, and mask secrets. Without it, a split config turns every surprise into an investigation across five files.

Also validate the merged result, not the fragments: a fragment is incomplete by design, and only the merged document has all the required keys. Run the schema check on the output of the merge, in CI and at start-up (validating config YAML with JSON Schema).

When to split, and how big

Split when the file has natural seams: environments, subsystems, teams. Do not split by size alone. A 300-line file with clear sections and anchors for shared parts is often easier to read than twelve files that must be opened together. Signs that splitting will help: frequent merge conflicts in one file, different owners for different sections, per-environment differences that are copied and pasted. Signs that it will not: the sections reference each other constantly, or you would need a tool to understand the result.

Keep fragments small, one concern each, with a comment at the top that says what it is and where it sits in the order. Use the same key names and nesting in every fragment so the merge lines up. And keep the number of layers low: base, environment and local override are enough for most systems.

Related reading

The cluster hub is structuring YAML configuration files that last. Related: environment specific YAML config without copy and paste and using anchors to reduce repetition in YAML config. The merge itself is in merging YAML files from the command line. The conf.d convention is described in many Unix packages; PyYAML's loader extension points are in its documentation, and Docker Compose's own include mechanism in the Compose include reference. Reformat a fragment in the YAML formatter.

When splitting a config makes it worse

The pieces depend on each other. If understanding one file requires reading three others, you moved the complexity and added file hops.

Only one consumer exists and the file is not changing. A stable file nobody edits does not need an architecture.

The tool has its own include. Use it, so validators and editors understand the structure.

You cannot print the effective configuration. Without that, debugging a split config costs more than the split saves. Build the printout first.