Converting JSON to YAML Without Losing Types or Quotes

JSON is a subset of YAML for most documents, so converting JSON to YAML sounds trivial: print the same data with less punctuation. The risk is that YAML is more eager to interpret what it reads. A string that looks like a boolean, a number or a date has to be quoted on the way out, or the next reader gets a different value than the one you wrote. This post converts a dictionary full of such strings, shows what a YAML library does, and what happens when you build the text yourself. It is part of the cluster on converting YAML to and from JSON, CSV, XML and Excel, and the other direction is converting YAML to JSON in Python.

A document built to cause trouble

This dictionary has twelve values chosen because YAML would read them as something other than strings if they were written plainly:

data = {
    "yes": "yes", "ver": "3.10", "zip": "02134", "n": "null",
    "d": "2024-01-01", "empty": "", "t": True, "num": 3.1,
    "tilde": "~", "colon": "a: b", "multi": "line1\nline2", "uni": "café",
}

Every value is a string except t (a boolean) and num (a float). Dumping it with a YAML library:

import yaml
out = yaml.safe_dump(data, sort_keys=False, allow_unicode=True)
print(out)

'yes': 'yes'
ver: '3.10'
zip: '02134'
n: 'null'
d: '2024-01-01'
empty: ''
t: true
num: 3.1
tilde: '~'
colon: 'a: b'
multi: 'line1

  line2'
uni: café

The library quoted exactly the values that needed it: 'yes' as both key and value, '3.10', '02134', 'null', '2024-01-01', the empty string, '~' and 'a: b'. It left true and 3.1 plain because those are the real boolean and the real float. It also folded the two-line string into a quoted scalar with a blank line standing for the newline. Loading the text back and comparing:

roundtrip equal: True

That is the property you want from a converter. Whatever type went in comes out, and a program reading the YAML gets the same dictionary a program reading the JSON got.

The same data written by hand

Now build the text the way a quick script would, one key: value line per entry, skipping the multiline string and the colon value, which we will come back to:

naive = "\n".join(f"{k}: {v}" for k, v in data.items() if k not in ("multi", "colon"))

Loading that and listing the entries that differ from the original:

{True: (True, 'bool'), 'ver': (3.1, 'float'), 'zip': (1116, 'int'), 'n': (None, 'NoneType'), 'd': datetime.date(2024, 1, 1), 'empty': (None, 'NoneType'), 'tilde': (None, 'NoneType')}

Reading the damage:

• The key yes became the boolean True.

• The version 3.10 became the float 3.1.

• The zip code 02134 became the integer 1116, because a leading zero marks an octal number in YAML 1.1.

• The string "null" became None.

• The date string became a date object.

• The empty string and ~ both became None.

Seven of ten values changed, silently, and the file is valid YAML. Then the value with a colon:

ScannerError mapping values are not allowed here

The line colon: a: b is a parse error. A library handles all of these by quoting, because it knows the rules of the target format. A string formatter knows none of them. If you are tempted to generate YAML with templates, read YAML strings, quoting and escapes first and then use a library anyway.

Options that shape the output

yaml.safe_dump has defaults that suit a machine more than a reader. Four options change that.

**sort_keys=False.** PyYAML sorts keys alphabetically by default. Pass False to keep the order the JSON had, which matters when the order carries meaning, such as a name first in each record.

**allow_unicode=True.** Without it, café is written as "caf\xE9", escaped and quoted. With it the text is readable, and the file must then be saved as UTF-8.

**default_flow_style=False.** Recent PyYAML versions already default to block style for nested collections. Setting True produces the compact {k: [0, 1, 2], s: {a: 1}} form, which is JSON-like and useful for small values you want to keep on one line.

**width.** Long strings are folded at 80 columns. A string of thirty repeated words was written as a quoted scalar continued over two lines; with width=200 it stayed on one:

long: 'word word word word word word word word word word wor

Folding changes the layout and not the value, so a loader returns the same string either way. For a long string, you may prefer a literal block (|), which PyYAML does not choose by itself; if you want that, use ruamel.yaml and a literal scalar type, or accept the folded output.

From the command line

yq converts a JSON file when you name both formats. A bare yq -P on a .json file keeps the JSON layout, because it picks the output format from the input, so say what you mean:

yq -p=json -o=yaml config.json

On a small file holding "yes": "yes", "3.10", "02134", null, the string "null" and true, it wrote:

yes: yes
ver: "3.10"
zip: "02134"

yq quoted the version and the zip code but left yes: yes bare. That is correct YAML 1.2, where yes is a plain string, and wrong for a YAML 1.1 reader: PyYAML, and therefore dbt and many other tools, would load the key and the value as True. The PyYAML dump above quoted both, so it is safe for either reader. Pick the converter by the consumer, and when you do not know the consumer, use the one that quotes more. The command-line path is covered in converting YAML to JSON with yq, and the YAML to JSON converter in the browser handles both directions.

Checking the result

Three checks catch most problems after a conversion.

1. Round trip. Load the YAML, dump it to JSON and compare with the original: json.loads(original) == yaml.safe_load(converted). The two must be equal.

2. Type scan. Walk the loaded data and flag any value whose type changed, or whose string form looks like a boolean, number or date. In the output above, every quoted value would be flagged for a human to confirm.

3. Lint the file. yamllint catches syntax and style issues, and a schema check catches type issues if you have a schema.

Quoting style when the consumer is human

The library's quoting is correct, and it can still be ugly. A generated file where every string is single-quoted reads worse than one a person wrote. Two choices help. Use ruamel.yaml, which keeps quotes you give it and supports block scalars, when the output will be edited by hand later. And for values that are plain text, leave the quotes to the library; do not post-process the output with a regular expression that strips quotes, since that is how '02134' loses its protection.

If humans will keep editing the file, hand over the first version and then treat it as a YAML file from that point on. Generated files should be regenerated, not edited; edited files should not be regenerated. Mixing the two is where comments go missing, as shown in round tripping YAML without losing comments.

When converting JSON to YAML adds no value

The consumer is a program. If only a machine reads the file, JSON is unambiguous and every parser agrees on it. YAML adds interpretation risk and no benefit.

The data is very large. YAML parsers do more work per byte than JSON parsers, and a multi-megabyte file converts into a file that is slower to load; large YAML files and parsing speed has the measurements. Keep large data in JSON, JSON Lines or a columnar format.

The values are mostly strings that look like other types. Identifiers, zip codes and codes of every kind generate a sea of quotes. The file is correct and noisy; ask whether the readers gain anything.

You need comments or anchors to appear. A conversion cannot invent comments. If you want them, add them by hand after converting and stop regenerating.

PyYAML documents the dump options in its reference page, and the rules for plain scalars, which explain every quote above, are in the YAML 1.2.2 specification. The boolean case is the subject of the Norway problem and YAML boolean surprises.