Validating Config YAML With JSON Schema in CI Pipelines

A linter tells you the YAML is well-formed. A schema tells you it says the right things: the keys exist and are spelled correctly, the values have the right types and fall in the right sets. JSON Schema is the standard way to write that down, and it works on YAML because YAML parses into the same data model as JSON. This post builds a schema for a small configuration, runs it in a way that fits CI, and shows the one thing most guides skip: the validator parses your YAML with its own rules, and those may not be the rules your application uses.

A schema

A configuration with an environment, a server and a features list:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "required": ["env", "server"],
  "additionalProperties": false,
  "properties": {
    "env": {"enum": ["dev", "staging", "prod"]},
    "server": {
      "type": "object", "required": ["host", "port"], "additionalProperties": false,
      "properties": {
        "host": {"type": "string"},
        "port": {"type": "integer", "minimum": 1, "maximum": 65535},
        "tls": {"type": "boolean"}
      }
    },
    "features": {"type": "array", "items": {"type": "string"}, "uniqueItems": true}
  }
}

Two settings do most of the work. additionalProperties: false turns unknown keys into errors, which catches typos. required makes missing keys errors, which catches deletions. Types, enum, minimum, maximum, pattern and uniqueItems cover values.

Running it

check-jsonschema --schemafile schema.json config.yaml

On a good file the output is ok -- validation done and the exit status is 0. On a file with several mistakes:

env: production
server:
  host: a.example
  port: "443"
  tsl: true
features: [search, search]
retries: 3

Schema validation errors were encountered.
  bad.yaml::$: Additional properties are not allowed ('retries' was unexpected)
  bad.yaml::$.env: 'production' is not one of ['dev', 'staging', 'prod']
  bad.yaml::$.server: Additional properties are not allowed ('tsl' was unexpected)
  bad.yaml::$.server.port: '443' is not of type 'integer'
  bad.yaml::$.features: ['search', 'search'] has non-unique elements

Five errors, all reported at once, each with a path ($.server.port). That is far better than failing at the first and fixing one at a time. The misspelled tsl shows why additionalProperties matters: without it, tsl: true is accepted and the server runs without TLS while the author believes otherwise. Exit status is nonzero on any error, which is what CI needs, and you can pass several files in one call.

The loader decides what is checked

Here is the part to understand before trusting a green result. check-jsonschema parses YAML itself (with YAML 1.2-style rules, in the tests here), then validates the data. Your application may use PyYAML, which implements 1.1. Take a config that has three of the classic ambiguities:

ver: 3.10
country: NO
ports: [80:80, 22:22]

and a schema that says ver is a string matching ^[0-9]+\.[0-9]+$, country is a string of two capitals and ports is a list of strings. The validator's verdict:

-- ver_bad.yaml
Schema validation errors were encountered.
  ver_bad.yaml::$.ver: 3.1 is not of type 'string'
exit=1

One error: the version loaded as the number 3.1 in every YAML version, so the schema caught it. But country: NO passed, because this loader reads it as the string NO, and the ports passed as strings. What PyYAML sees in the same file:

{'ver': 3.1, 'country': False, 'ports': ['80:80', 1342]}

country is False and one port is the integer 1342. A PyYAML-based application receives bad data from a file that the schema check approved. The quoted version, ver: "3.10", country: "NO" and quoted ports, passes in both.

The lesson is a rule, not a defect: validate with the parser your application uses, or make the file safe for both by quoting. Options: run the validation in Python, loading with the same function the application calls and passing the result to jsonschema (jsonschema.validate(instance, schema)); or use a loader test next to the schema check; or ban unquoted ambiguous scalars with yamllint's truthy and quoted-strings rules (linting YAML with yamllint). Do at least one of them, or a schema check gives false confidence about the YAML 1.1 problems.

Schemas from models

If the application already has a typed model, generate the schema from it instead of maintaining two descriptions. With pydantic:

import json
open("gen_schema.json", "w").write(json.dumps(C.model_json_schema()))

and check a bad file with it:

gen_bad.yaml::$: Additional properties are not allowed ('extra' was unexpected)
gen_bad.yaml::$.env: 'test' is not one of ['dev', 'prod']
gen_bad.yaml::$.server: 'port' is a required property

The model's Literal became an enum, extra="forbid" became additionalProperties: false and the missing field became required. Generate the schema in CI from the code, commit it if editors need it, and validate configuration files with it. The model and the schema cannot drift, because one produces the other (validating YAML configuration with pydantic). For non-Python consumers, a hand-written schema is the shared contract.

Putting it in CI

A job runs the check on every configuration file, on the merged configuration for each environment, with pinned tool versions:

config-check:
  steps:
    - run: pip install check-jsonschema==0.39.0
    - run: check-jsonschema --schemafile schemas/app.json config/*.yaml
    - run: python tools/merge_env.py prod > /tmp/prod.yaml && check-jsonschema --schemafile schemas/app.json /tmp/prod.yaml

The first validates the files as committed; the second validates the effective production configuration after the layers are merged, which is what actually runs (environment specific YAML config without copy and paste). Fragments of a layered config are incomplete by design and fail required, so validate only the merged output.

For editors, associate the schema with the file so mistakes are underlined as you type. Many YAML language servers read a comment at the top of the file, # yaml-language-server: $schema=./schemas/app.json, or a workspace setting. Combined with the CI check, authors see errors before the commit and the pipeline enforces them after.

What a schema cannot say

Schemas check structure and simple constraints. They cannot check that a port is free, that a host resolves, that two fields agree with each other in complex ways, or that a referenced secret exists. They cannot see comments. And uniqueItems compares whole items, so two records with the same name and different other fields pass. For cross-field rules, use a typed model with validators, or a test. Keep the schema for the shape and the tests for the meaning.

Keeping the schema alive

A schema is code, and like code it rots without ownership. Three habits keep it useful. Version it with the application: the pull request that adds a setting adds the schema entry and the example. Test it with fixtures, a directory of valid files that must pass and invalid files that must fail with a particular message, so that tightening a rule cannot silently loosen another. And publish it where authors work, so the editor can offer completion and underline mistakes as they type.

When you add a required key, think about existing files. A new required entry fails every environment file that predates it. Introduce it as optional with a default, deploy the code that handles both, update the files and then make it required. When you remove a key, keep it in the schema as deprecated for a release (a description that says so) before forbidding it.

Finally, give the schema's error messages a human wrapper in the pipeline. The raw output is precise and terse. A comment on the pull request that lists the failing paths and links to the configuration guide turns a red check into something an author can fix without asking.

Related reading

The cluster hub is structuring YAML configuration files that last. Related: validating YAML configuration with pydantic, YAML 1.1 versus 1.2 for the loader question and naming keys consistently in YAML config, which uses a schema to enforce key style. The specification is at json-schema.org, and the tool at check-jsonschema. Preview a config's types in the YAML to JSON tool.

When schema validation will miss a problem

The validator parses YAML differently from your application. Quote ambiguous values or validate through the application's loader.

The rule involves several fields or the outside world. Use a model with validators and start-up checks.

The file is a template. Validate the rendered output.

The schema allows extra keys. Without additionalProperties: false, typos pass. Set it, and extend the schema when a key is added on purpose.