Using JSON in Bash Scripts Safely With jq Arguments

Shell scripts end up handling JSON all the time: a CI step reads the output of a cloud CLI, a deploy script builds a request body, a cron job reads a config. The shell has no types and no escaping rules that match JSON, so the usual pitfalls are quoting and splitting. The tool for nearly all of it is jq, and most of the safe ways to use it are a matter of which of its options you reach for.

I ran each case in a container and kept the results. The problems are the ones that show up on the first day a value contains a space, a quote or a newline.

Never build JSON with printf

The obvious way to make a JSON body in a script is a template:

name='Ana "The DBA" O'"'"'Neil'
printf '{"name":"%s"}' "$name" | jq -c .
# parse error: Invalid numeric literal at line 1, column 18

The quotes in the name ended the string early and the result is not JSON. The error is the good case. The dangerous case is a value that produces valid JSON that means something else:

role='user", "admin": true, "x":"'
printf '{"role":"%s"}' "$role" | jq -c .
# {"role":"user","admin":true,"x":""}

The value added a member. If role comes from a branch name, a form field or an API response, that is injection into a JSON body, the same bug as SQL built from strings (JSON injection in SQL built from request bodies shows the database version).

jq builds the document and does the escaping:

jq -nc --arg name "$name" --arg note "$note" '{name:$name, note:$note}'
# {"name":"Ana \"The DBA\" O'Neil","note":"line1\nline2\ttab"}

jq -nc --arg role "$role" '{role:$role}'
# {"role":"user\", \"admin\": true, \"x\":\""}

The quotes, the newline and the tab were escaped, and the hostile role stayed one string. This is the rule that covers most of the others: values go in as arguments, never into the text of the program or the document.

Strings, numbers and booleans

--arg always makes a string. --argjson parses its value as JSON:

jq -nc --arg n 5 --argjson m 5 --argjson t true '{n:$n, m:$m, t:$t}'
# {"n":"5","m":5,"t":true}

A number passed with --arg becomes the string "5", which an API expecting a number will reject or coerce. --argjson with text that is not JSON fails at once: jq: invalid JSON text passed to --argjson, so a variable that should hold JSON but holds abc stops the script before it posts anything. For many values, --args puts the rest of the command line in $ARGS.positional:

jq -nc '$ARGS.positional' --args a "b c" 'd"e'
# ["a","b c","d\"e"]

Environment variables work too, which is a good way to pass a secret without it appearing in the process list: export TOKEN=... and then jq -nc '{t: env.TOKEN}' gave {"t":"s3\"cr3t"}. Arguments to jq are visible in ps; the environment is less exposed.

Reading values into the shell

The classic mistake is a for loop over command output:

for n in $(jq -r '.names[]' n.json); do printf '[%s]' "$n"; done
# [Ana][Lee][Bo][Cy][Dee]

The names were Ana Lee, Bo and a value with an embedded newline, Cy\nDee. The unquoted substitution split on every space and newline and produced five words. A while read loop is better but still splits on newlines:

jq -r '.names[]' n.json | while IFS= read -r line; do printf '[%s]\n' "$line"; done
# [Ana Lee]
# [Bo]
# [Cy]
# [Dee]

Spaces survived, and the embedded newline made the last value two items. mapfile -t arr < <(jq -r '.names[]' n.json) gave four array elements for the same reason. Values that contain newlines need a different separator. jq -j prints without newlines, so a NUL byte can be the separator, and read -d '' reads up to a NUL:

jq -j '.names[] + "\u0000"' n.json | while IFS= read -r -d '' x; do printf '<%s>' "$x"; done
# <Ana Lee><Bo><Cy
# Dee>

All three came through intact, with the newline inside one value. Use this when the data is not under your control. When you know the values are simple (version numbers, hostnames), mapfile is fine. For a handful of fields from one object, jq -r '[.a,.b]|@tsv' or the @sh formatter that quotes for the shell are alternatives, with their own care needed for tabs and quotes.

Null, false and exit codes

jq is not strict about what counts as success unless you ask:

v=$(echo '{"a":null}' | jq -r '.a');             # [null]
v=$(echo '{"a":null}' | jq -r '.a // empty');    # []

-r printed a JSON null as the four characters null. A script that tests [ -n "$v" ] thinks it has a value. The alternative operator // empty turns null and false into nothing. To test for presence, use the exit status with -e:

echo '{"a":null}'  | jq -e '.a'   -> 1
echo '{"a":false}' | jq -e '.a'   -> 1
echo '{"a":1}'     | jq -e '.b'   -> 1   (missing key)
echo '{bad'        | jq .         -> 4   (invalid input)
echo '{"a":1}'     | jq -e '.a'   -> 0

With -e the exit status is 1 if the last output is false or null (or there is none), 0 otherwise, and 4 for a parse error. Note that -e cannot tell false from missing. In a pipeline, set -o pipefail makes the failure visible: false | true returned 0 without it and 1 with it, and a jq parse error at the front of a pipe is otherwise masked by the command at the end. Use set -euo pipefail at the top of scripts that handle API output.

Big numbers and the program text

jq 1.6 converts numbers to doubles. A 16 digit id came out changed:

echo '{"id":9007199254740993}' | jq -c .id            # 9007199254740992
echo '{"id":"9007199254740993"}' | jq -c .id          # "9007199254740993"

The first line lost the last digit. jq 1.7 and later preserve the original text of a number literal that is passed through unchanged, but arithmetic still goes through doubles. If you handle ids, either keep them as strings, or check the version you run in CI.

The last pitfall is the filter itself. Putting a shell variable into the program text is injection into jq:

export SECRET_TOKEN=abc123
field=".a, env.SECRET_TOKEN"
jq -nc "{a:1} | $field"
# 1
# "abc123"

The "field name" smuggled in an expression that read an environment variable. If the field comes from outside, pass it as data: jq -nc --arg f a '{a:1} | .[$f]' printed 1 and could do nothing else. Keep the program in single quotes, and use $var for anything dynamic.

Habits that keep scripts honest

Validate a response before using it: jq -e . >/dev/null fails on non-JSON, which is how an HTML error page from a proxy is caught. Check the HTTP status separately (curl --fail-with-body), because a curl that returns a 500 with a JSON error body is still successful JSON. Quote every expansion. Put the jq programs in files for anything longer than a line, so they are testable. For anything with real logic, a Python script is easier to test than a Bash one (choosing between jq, JSONPath and JMESPath and jq versus Python cover when to switch), and security-sensitive input deserves the checks in JSON security mistakes in apps and pipelines.

The manual is at jqlang.github.io/jq/manual, and the Bash behaviors above are described in the GNU Bash manual. Use the JSON formatter to check a body before you send it.

When a shell script is the wrong place for JSON

The logic has branches. Loops over nested data, retries and error handling become hard to read in Bash. Move to Python or Node once the script passes thirty lines.

Secrets are involved. Passing tokens through arguments puts them in the process list. Use environment variables or files with tight permissions, and keep them out of logs.

The data is large. Streaming with jq --stream or JSON Lines is possible, and a database or a proper tool is better (handling JSON files too large to parse).

Numbers must be exact. Use a language with arbitrary precision integers, or keep numbers as strings through the whole path.