Using jq to Slice Up JSON in the Terminal Reliably

jq is the tool I reach for first when someone hands me a JSON file and asks what is in it. It is a small, fast program with a language of its own, and the language is the hard part: filters, pipes, generators and the way one input can produce zero, one or many outputs. Once that model clicks, most jobs are one line. This guide is the map for the series. It covers installing and versions, the mental model, selecting and reshaping, grouping, CSV, JSON Lines, passing values safely from the shell, editing files, debugging, memory and speed, and where jq is the wrong tool. Every number and message below comes from running jq 1.8.2 against the data used throughout this series: 20,000 orders, 200,000 events, and a 100,000 line application log.

Installing and the version that matters

Most Linux distributions ship an old jq. The Debian package I had installed reported jq-1.6, and the current release at the time of writing was 1.8.2. The difference is not cosmetic. Four things changed that you will notice, and the first one can corrupt data:

input                           jq 1.6                   jq 1.8.2
12345678901234567890            12345678901234567000     12345678901234567890
[1.0, 1e3, 0.10]                [1,1000,0.1]             [1.0,1E+3,0.10]
pick(.a), abs, trim, ltrimstr   pick, abs, trim missing  available (ltrimstr on a number errors)
limit(0; 1,2)                   [1]                      []

Version 1.6 converts every number to a 64-bit float, so a large identifier is silently rounded on the way through. Version 1.7 and later keep the original text of a number literal until you do arithmetic on it. That is a real fix for pass-through filters: jq . on a file with 64-bit ids now returns them untouched. It has a side effect that looks odd the first time. A number is printed as it was written, so 1.0 stays 1.0 and 0.10 stays 0.10, and 1e3 is printed as 1E+3, which is valid JSON and not what you typed. As soon as you compute with it, the value becomes a double: [12345678901234567890, 1.0, 1e3] | map(. + 0) gave [12345678901234567000,1,1000]. If ids matter, treat them as strings or do not do arithmetic on them. The installation options and how to check which jq you have are in installing jq and running your first filter.

The mental model

A jq program is a filter. It takes one input value and produces a stream of zero or more outputs. Everything else follows from that:

• . is the identity filter. .name produces the value of a key, .[] produces every element of an array (or every value of an object), and .[2:5] produces a slice.

• The pipe a | b runs b on every output of a. The comma a, b produces the outputs of both. They bind differently: .data.Items[0] | .id, .Name gave 1 then "a", because the comma is inside the right-hand side of the pipe.

• A missing key gives null, not an error: .[0].customer.phone printed null. That is convenient and the source of most "my filter returns nothing" confusion.

• Iterating over null is an error (Cannot iterate over null (null)), and indexing the wrong type is an error (Cannot index array with string ("id")). The ? suffix turns an error into no output, and // supplies a default.

The command-line options that change how input and output work are few, and I use them in this order of frequency: -r (print strings without quotes), -c (one value per line), -n (start with null and no input), -s (slurp all inputs into one array), -e (set the exit status from the last output), --arg and --argjson (pass values in), and -S (sort keys). From the orders file:

$ jq length orders.json
20000
$ jq -c '.[0] | keys' orders.json
["customer","id","items","paid","placed"]
$ jq -r '.[0].customer.name' orders.json
Customer 1
$ jq '.[0].customer.name' orders.json
"Customer 1"

The last two lines differ only by -r, and forgetting it is why a shell variable ends up holding quote characters. For the full, ordered walk through these, start with installing jq and running your first filter.

Selecting fields and filtering arrays

The two jobs I do most are "give me these fields" and "give me the records where". Both are a pipe away:

jq -c '.[:2][] | {id, city: .customer.address.city, n: (.items | length)}' orders.json
# {"id":1,"city":"Oslo","n":2}
# {"id":2,"city":"Lima","n":3}

jq '[.[] | select(.paid)] | length' orders.json                    # 6666
jq -c '[.[] | select(any(.items[]; .price == 0)) | .id]' orders.json  # [9000,18000]

{id, city: ...} is object construction with a shorthand for {id: .id}. select(f) passes its input through when f is truthy and produces nothing otherwise, which is how filtering works. any(generator; condition) asks whether any element qualifies; it found the same two zero-price orders (9000 and 18000) that the Python and JavaScript articles in this series found. Combining conditions uses and, or and not: select(.customer.address.city == "Lima" and (.items | length) > 3) matched 1,666 orders. The details, including //, the ? operator and the difference between == and =, are in selecting fields and filtering arrays with jq. That last difference bites: select(.id = 1) is an assignment, which is truthy, so it selects everything and rewrites id.

Reshaping

Reshaping builds new objects from old ones. {order_id: .id, customer_name: .customer.name} renames; del(.items, .customer.address) removes; .paid |= not updates in place; with_entries(.key |= "c_" + .) renames every key of an object; walk(...) applies a function everywhere. To drop all the nulls from a record with nested objects, del(.. | nulls) is enough and gave {"id":1,"ts":...,"user":{...},"event":"click","tags":[...],"props":{...}} with amount removed. To flatten a nested object to dotted keys:

head -1 events.ndjson | jq -c '[paths(scalars) as $p | {key: ($p | map(tostring) | join(".")), value: getpath($p)}] | from_entries'
# {"id":1,"ts":"2026-10-02T01:01:00Z","user.id":1,"user.plan":"pro","event":"click","tags.0":"sale","tags.1":"eco","props.page":"/p/1","props.ms":1}

paths(scalars) enumerates the path to every leaf, which makes it the most useful single filter for exploring an unknown document. The patterns for renaming, merging, defaults and nested updates are in reshaping JSON objects with jq. The same flatten operation is available in the browser with the flatten and unflatten JSON tool.

Grouping and counting

group_by(f) sorts by f and returns an array of arrays, one per distinct value. Counting records per city is:

jq -c 'group_by(.customer.address.city) | map({city: .[0].customer.address.city, n: length})' orders.json
# [{"city":"Lima","n":6667},{"city":"Lyon","n":6666},{"city":"Oslo","n":6667}]

For large inputs I prefer reduce, which does not sort and does not hold the group arrays: reduce .[] as $o ({}; .[$o.customer.address.city] += 1) gave the same counts as an object. The same reduce works on a stream of events (jq -nc 'reduce inputs as $e ({}; .[$e.user.plan] += 1)' returned {"pro":66667,"team":66667,"free":66666}), and sums work by changing += 1 to += ($e.amount // 0). Revenue per city came out as {"Oslo":2126049.9399998733,...}: the trailing digits are ordinary floating point addition and not a bug in jq, and the same noise appears in every language, as the 2494844.5600000042 sum of event amounts did here (Python's sum on a list gave 2494844.56 because recent versions use compensated summation). Round when you print. More patterns, with ordering and top-N, are in grouping and counting JSON records with jq.

JSON to CSV

@csv and @tsv format an array as a row, so the job is to turn each record into an array of the columns you want, with the header first:

jq -r '(["id","city","paid","placed"]), (.[:4][] | [.id, .customer.address.city, .paid, .placed]) | @csv' orders.json
# "id","city","paid","placed"
# 1,"Oslo",false,"2026-02-02"
# 2,"Lima",false,"2026-03-03"

Strings are quoted, numbers and booleans are not, and null becomes an empty field, which is indistinguishable from an empty string in the result ([{"a":null},{"a":""}] produced an empty line and "", which most CSV readers treat alike). @csv on an object fails with object (...) cannot be csv-formatted, only array, and on a nested array with array ([2]) is not valid in a csv row, so nested values need tojson or flattening first. A cell containing a comma, a quote or a newline is quoted correctly ("say ""hi""", and a two-line cell stays inside its quotes), but a value that starts with =, + or @ is written as is, so a spreadsheet will evaluate it as a formula; sanitize anything that came from outside. Ragged records need a header built from the union of keys. All of this is in turning JSON arrays into CSV with jq, and the opposite direction lives in the CSV articles of this site.

JSON Lines

NDJSON is jq's natural input: each line is a complete value, and jq runs your filter on each in turn. jq -c 'select(.level=="error")' applog.ndjson printed the 4,970 error lines among 100,000 in 0.19 s. Two details separate a good log filter from a bad one. First, a corrupt line stops the whole run: with an unfinished record in the middle of a file, jq printed the first line, then jq: parse error: Unfinished JSON term at EOF and exited with status 5, and the lines after the break never appeared. To skip bad lines, read raw and parse yourself: jq -R -c 'fromjson?' printed the two good lines of my test file and dropped the bad one. Second, blank lines are fine, and CRLF line endings parse correctly, because whitespace between values is allowed. The log recipes (top errors, latency percentiles, counts per minute) are in using jq to read logs written as JSON Lines, and the format itself is in JSON Lines for data pipelines and log files.

Passing values from the shell

This is where jq scripts become security bugs. The tempting way to use a shell variable is to splice it into the filter string:

city='" or ($ENV.SECRET_TOKEN|length>0) or "'
jq -c "[.[]|select(.customer.address.city==\"$city\")]|length" orders.json
# 20000

The value was not data. It closed the string, added a clause that read an environment variable and re-opened the string, so the filter selected all 20,000 orders. A value with a plain double quote (Os"lo) merely produced a syntax error, and a more careful one reads the environment or rewrites the filter. The correct form passes the value separately:

jq -c --arg c "$city" '[.[] | select(.customer.address.city == $c)] | length' orders.json
# 0

--arg name value binds $name as a string, whatever it looks like (--arg n 5 gave "5"), --argjson parses its value as JSON (--argjson n 5 is the number 5, and invalid JSON fails with invalid JSON text passed to --argjson), --args and --jsonargs collect positional values into $ARGS.positional, and $ENV.NAME reads the environment (null when unset). Note that test(), sub() and friends treat their argument as a regular expression, so --arg p 'a.c' matches abc; use contains or index for literal text. Arguments are visible in the process list, so a secret should travel through the environment or --rawfile. The full set of cases, plus getting values back out of jq into the shell safely with -r, @sh and --raw-output0, is in passing shell variables into jq safely.

Editing files

jq has no in-place flag. The first thing everybody tries is the one that destroys the file:

jq '.version = "2.0"' cfg.json > cfg.json     # cfg.json is now 0 bytes

The shell truncates the output file before jq reads it. sponge from moreutils soaks up all of its input before writing, which fixes that case, but it has a trap of its own: when jq failed (a typo in the filter), the empty output was still written to the file, which ended at 0 bytes, even with set -o pipefail in effect, because the pipeline's exit status arrives after the damage. The pattern I use is a temporary file and a rename, with the && ensuring the rename only happens on success:

jq '.version = "2.0"' cfg.json > cfg.json.tmp && mv cfg.json.tmp cfg.json

When the filter was broken, the original stayed at 129 bytes. Two behaviors to know: mv changed a mode 600 file to 644 (the new file takes the default mode, where sponge preserved 600), and replacing a symlink with mv turned the link into a regular file while leaving the target unchanged. Reformatting is a side effect of any edit, since jq re-indents the whole file, 1e3 becomes 1E+3, and escapes like é come out as the real character. All of it is in modifying JSON files in place with jq and sponge.

When the filter returns nothing

Most jq debugging is one of five things. The key is spelled differently (.data.items printed null because the key is Items). The value is the wrong type (select(.id == "1") printed nothing because the id is a number). The key needs quoting (.data.my-key is parsed as subtraction and fails with key/0 is not defined; write .data."my-key"). The input is NDJSON and the filter expects an array (.[0] against each line gave Cannot index object with number (0) repeatedly, and -s fixes it). Or the filter is right and select removed everything, which exits with status 4 under -e. The tools for finding out which: keys, type, paths, debug (it prints ["DEBUG:", value] to stderr and passes the value through) and getpath. The worked cases are in debugging a jq filter that returns nothing.

Memory and speed

jq is fast to start (the whole process took under 10 ms) and fast on streams, and its memory use depends entirely on how you read. On the 36 MB, 200,000 event file:

jq -s 'map(.amount // 0) | add'                          1.0 to 1.5 s   459 MB
jq 'map(.amount // 0) | add'  (one big array file)        1.0 to 1.1 s   460 MB
jq -n 'reduce inputs as $e (0; . + ($e.amount // 0))'     0.6 to 0.8 s     2 MB
jq -n --stream (event array, streaming parser)            2.8 s            2 MB

Slurping holds everything and costs about twelve times the file size. The -n plus inputs form handles one value at a time and used 2 MB. Under a container limit of 120 MB, the slurped and whole-array runs were killed (exit status 137 with no message), while the inputs and --stream runs finished with the same total. --stream turns a single huge document into path and value events, which is the only way to process one 5 GB array, at a cost of being slow to write and about four times slower to run. If you control the producer, use one record per line.

Against Python on the same jobs the picture is mixed. Summing amounts over NDJSON took 0.62 s in jq against 0.50 to 0.65 s in a Python loop. Selecting errors from the log took 0.19 s against 0.22 to 0.25 s. Grouping and counting took 0.45 s in jq and 0.21 to 0.25 s in Python with a Counter. Startup favored jq (about 5 ms against 50 ms with import json). The summary is that jq is competitive for filters and not for aggregation, and the comparison with the code is in jq versus Python for quick JSON jobs. A trick worth knowing: prefilter with grep when the pattern is textual. grep '"level": "error"' applog.ndjson | jq -c . took 0.02 s against 0.19 s, because grep rejects 95 percent of the lines before jq parses them. It is valid only when the text pattern cannot match anything but the intended lines, so use it for speed, not for correctness.

Exploring a document you have never seen

Before writing any filter, find out what is in the file. I use four filters for that, in this order. keys shows the top-level names (jq -c '.[0] | keys' orders.json gave ["customer","id","items","paid","placed"]), map_values(type) shows what kind of value each one holds ({"data":"object","count":"string"} for a test document, which is how I noticed a count stored as text), [paths(scalars)] lists every leaf, and [.. | objects | select(has("Name")) | .id] finds objects anywhere in the tree that have a key. The recursive descent operator .. visits every value at every depth, and combining it with select is the quickest way to answer "where in this document does the word appear". To find the path to a value you remember, [paths(. == "b")] returned [["data","Items",1,"Name"]], which you can feed straight back into getpath or into an assignment. For a document too big to read, tostream prints each leaf as a path and a value on its own line, and limit(5; tostream) stops after five, which is a cheap way to sample the structure of a file you do not want to load.

The same habit helps with unfamiliar JSON Lines. Look at one line first with head -1 file | jq ., then at the key set across a sample with jq -nc '[limit(1000; inputs) | keys] | add | unique', because records in a long file rarely all have the same keys, and the union is the schema you actually have. The JSON formatter does the first step in a browser if jq is not installed.

Strings, numbers and dates

jq's string functions cover most cleaning jobs, and a few behave in ways worth knowing. split(" ") and join("-") are literal, while test, match, sub, gsub and splits use regular expressions, so --arg p 'a.c' with test($p) matched abc and with split($p) it did not. ascii_downcase, ltrimstr, startswith and ascii need string input and fail on anything else: under 1.8.2, 1 | ltrimstr("a") raised startswith() requires string inputs, where 1.6 passed the number through, so a filter that worked on mixed data in one version can fail in the next. tonumber is stricter than it was: "007" | tonumber gave 7, " 1" | tonumber raised an error under 1.8.2 (it returned 1 under 1.6), and "1e3" | tonumber printed 1E+3. Wrap it in try tonumber catch null when the input is not under your control, and the result [1, null] for ["1","x"] is the pattern to remember.

Dates are the part of jq that most people avoid, and the basics are small. fromdate parses an ISO 8601 UTC string to epoch seconds ("2026-10-26T12:30:00Z" | fromdate gave 1793017800), todate goes the other way, and strptime with mktime handles other formats ("2026-10-26" | strptime("%Y-%m-%d") | mktime gave 1792972800, which is UTC midnight). The format must match exactly and the zone is UTC; there is no time zone database in the basic functions, so local time conversion is a job for another tool. Because the ISO strings in a log sort lexicographically in time order, you often need no date functions at all: select(.ts >= "2026-10-26T01:00:00Z" and .ts < "2026-10-26T01:00:05Z") compares strings, and .ts[:16] slices the timestamp to the minute for grouping. That is the technique behind the per-minute error counts in the log article, and it is exact only because the format is fixed-width and zero-padded, so check that before relying on it.

Output control and exit codes

How jq writes matters when its output goes to another program. By default every output value is pretty printed with two spaces and followed by a newline. -c prints each value on one line, which is what you want for JSON Lines. --indent n (up to 7) and --tab change the indentation (--tab produced a real tab character), -S sorts keys, and -a escapes non-ASCII characters ("caf\u00e9 \ud83d\ude00"). For strings, -r drops the quotes and -j also drops the newline, so jq -j '.[0].id, " ", .[1].id' printed 1 2 with no trailing line break. If values can contain newlines, none of those is safe for a pipe into xargs; --raw-output0 separates outputs with a NUL byte, and jq --raw-output0 '.[:3][] | .customer.name' | xargs -0 -n1 printf '[%s]\n' handled names with spaces correctly.

Exit codes are how scripts know what happened, and jq uses them consistently. A clean run exits 0 whatever it printed, so jq '.[0].paid' returning false still exits 0. With -e the status reflects the last output: 0 for any value except false and null, 1 for those, and 4 when there was no output at all (select removed everything). An unreadable file exits 2, a JSON syntax error exits 5 (the same as a runtime error such as error or iterating null), and halt_error writes its input to standard error and exits with the code you choose ("s\n" | halt_error(3) exited 3, and a bare halt_error exits 5). That gives a script a way to branch: if jq -e '.[0].paid' orders.json > /dev/null; then ... runs the then branch only for a true answer and treats false, null and "nothing selected" alike as no. When the distinction matters, test the exit code explicitly. Because jq writes errors to standard error and keeps going to the next input where it can, a pipeline can print partial results and still exit non-zero, so check the status of the command, not just whether it printed something.

A cheat sheet to keep open

The operators and builtins I use weekly, one line each, are collected in a jq cheat sheet for everyday work. For the Python equivalents of these jobs see handling JSON in Python beyond the json module, and for SQL over the same files, querying JSON files with DuckDB. To format a document you have copied from somewhere, the JSON formatter needs no installation, and pretty printing JSON in Python and from the terminal covers the alternatives that ship with your system.

When jq is the wrong tool for the job

The job is aggregation over millions of rows. A Counter in Python, SQL in DuckDB or Miller will be shorter to write and often faster. jq's group_by sorts and holds everything.

The logic needs state, error handling or I/O. A jq program that grows past ten lines is a Python script in disguise, written in a language most colleagues cannot read. Write the script.

The file is one enormous document. Streaming mode works and is slow and awkward. Convert to JSON Lines once, then use ordinary filters.

Numbers must be exact decimals. jq uses doubles for arithmetic. Money should be summed in a database or a decimal library.

The input is nearly JSON. Comments, trailing commas and single quotes make jq fail with a parse error (fixing JSON that is really a JavaScript object literal). Repair first.

You need guarantees about the shape. jq will happily output null for a typo. Validate with a schema when a wrong answer is worse than no answer (validating JSON with jsonschema in Python).

The language is documented in the jq manual, the release notes for each version are on the jq GitHub releases page, and the interactive jq playground is useful for trying a filter against a snippet.