Text processing
jq
Query, filter and reshape JSON from the command line with a small expression language.
jq [OPTION...] FILTER [FILE...]
jq reads JSON, runs a filter over it, and writes JSON back out. The simplest filter, .,
passes the input through unchanged and pretty-prints it. .key reaches into an object,
.[0] into an array, and .[] produces each element of an array as a separate output. Filters
chain with | just like shell commands, so .servers[] | .name reads "for each server, its
name".
The model to keep in mind is that every filter takes one input and can produce zero, one or
many outputs. select(cond) passes its input through only when the condition is true, which
is how you filter; map(f) applies f to every element and keeps the result an array;
{name, ip} or [...] build new objects and arrays from pieces.
Strings come out with JSON quotes unless you pass -r (raw), which is what you want when the
result feeds a shell loop. Never build a filter by pasting a shell variable into it; pass it
with --arg (as a string) or --argjson (as JSON). For CSV or plain columns, see awk;
for fetching JSON from an API first, see curl.
Examples
Pretty-print JSON
echo '{"name":"web01","tags":["prod","eu"],"up":true}' | jq .
{
"name": "web01",
"tags": [
"prod",
"eu"
],
"up": true
} . is the identity filter: output equals input. jq re-indents it with two spaces and one
key per line. It also validates: malformed JSON is an error with a non-zero exit status.
Get one field
jq '.version' data/config.json
"2.4.1"
.version selects the key from the top-level object. The result is a JSON string, so it
keeps its double quotes.
Print strings without quotes
jq -r '.database.host' data/config.json
db01
Paths chain with dots: .database.host. -r prints strings raw, without JSON quotes, so
the value can go straight into a shell variable: host=$(jq -r .database.host config.json).
What happens with a missing key
jq '.database.user' data/config.json
jq '.database.user // "postgres"' data/config.json
null "postgres"
A key that does not exist gives null, not an error. The alternative operator //
substitutes a default when the left side is null or false.
Iterate over an array
jq -r '.[].name' data/servers.json
web01 web02 db01 cache01 build01
servers.json is an array. .[] emits each element separately, and .name runs on each
one, giving five outputs. With -r they print one per line, ready for while read.
Filter with select
jq -r '.[] | select(.cpus >= 8) | .name' data/servers.json
db01 build01
select(cond) passes an element through only when cond is true, otherwise drops it.
Only db01 and build01 have eight or more CPUs.
Filter on booleans and strings
jq -c '.[] | select(.up == false or (.tags | index("us"))) | {name, tags}' data/servers.json
{"name":"web02","tags":["prod","us"]}
{"name":"cache01","tags":["staging"]} Conditions combine with and, or and not. index("us") on an array returns the
position of that element or null, and null counts as false. {name, tags} is shorthand
for {name: .name, tags: .tags}. -c prints each object on one line.
Build new objects
jq -c '.[] | {host: .name, addr: "\(.ip):22", prod: (.tags | contains(["prod"]))}' data/servers.json
{"host":"web01","addr":"10.0.0.11:22","prod":true}
{"host":"web02","addr":"10.0.0.12:22","prod":true}
{"host":"db01","addr":"10.0.0.21:22","prod":true}
{"host":"cache01","addr":"10.0.0.31:22","prod":false}
{"host":"build01","addr":"10.0.0.41:22","prod":false} Object construction picks and renames fields. "\(...)" is string interpolation. The
prod value is computed: contains(["prod"]) tests whether the tags array includes that
string.
Transform every element with map
jq -c 'map(.cpus) | add, max, (add / length)' data/servers.json
34 16 6.8
map(.cpus) turns the array of servers into an array of numbers. The three filters after
it each receive that array: add sums it to 34, max finds 16, and dividing the sum by
length gives the average.
Count elements and list keys
jq -c 'length, keys, (.features | length)' data/config.json
6 ["app","database","debug","features","server","version"] 3
On an object, length counts its keys and keys lists them sorted alphabetically (use
keys_unsorted to keep file order). On an array, length counts elements.
Use a shell variable safely with --arg
role=web
jq -r --arg role "$role" '.[] | select(.role == $role) | "\(.name) \(.ip)"' data/servers.json
web01 10.0.0.11 web02 10.0.0.12
--arg role "$role" defines $role inside the filter as a JSON string. The filter itself
stays in single quotes, so the shell never touches it. Use --argjson when the value must
be a number or other JSON, for example --argjson min 8.
Turn JSON into tab-separated columns
jq -r '.[] | [.name, .ip, .role, .cpus] | @tsv' data/servers.json
web01 10.0.0.11 web 4 web02 10.0.0.12 web 4 db01 10.0.0.21 database 16 cache01 10.0.0.31 cache 2 build01 10.0.0.41 ci 8
Build an array per element, then @tsv formats it as one tab-separated line (@csv gives
quoted CSV). With -r the result is plain text that column -t, sort or awk can use.
Group and count
jq -c 'group_by(.role) | map({role: .[0].role, count: length})' data/servers.json
[{"role":"cache","count":1},{"role":"ci","count":1},{"role":"database","count":1},{"role":"web","count":2}] group_by(.role) sorts by role and returns an array of arrays, one per distinct role.
map turns each group into an object with the role and the group's size.
Change a value and write the result
jq '.server.port = 9090 | .debug = true' data/config.json > config.new
jq -c '{port: .server.port, debug}' config.new
{"port":9090,"debug":true} = sets a path and returns the whole modified document. jq never edits in place: write to
a new file and move it over the old one. Redirecting straight back into
data/config.json would truncate it before jq reads it.
Invalid JSON is an error
echo '{"port": 8080,}' | jq .port
jq: parse error: Expected another key-value pair at line 1, column 15
The trailing comma is not valid JSON. jq reports where parsing failed and exits with status
5, so set -e scripts and if jq ... tests notice the bad input.
Common pitfalls
Without
-r, strings keep their JSON quotes, and$(jq .name file)gives you"web01"with the quotes. Add-rfor anything that goes into a shell variable.Pasting variables into the filter (
jq ".[] | select(.name == \"$n\")") breaks on quotes and allows injection. Use--arg n "$n"and refer to$ninside the filter.jq . file > fileempties the file: the shell truncates it before jq starts. Write to a temporary file andmvit..foo.baron a value where.foois a string or number is an error (Cannot index string). Use.foo.bar?to suppress it or check the type first.Keys with dashes or starting with digits need quoting:
."audit-log"or.["audit-log"], because.audit-logmeans.auditminuslog.
GNU vs BSD and macOS
The examples above ran the GNU/Linux version. These are the differences that matter on macOS and the BSDs (written from their documentation, not run here).
jq is one implementation everywhere, so there is no GNU/BSD split, but versions differ: Debian 12 ships 1.6 and Debian 13 ships 1.7. Newer features such as
pick()and--raw-output0need 1.7; checkjq --versionbefore relying on them.
Options worth knowing
A selection, described in our own words. For the full list, see jq(1) or jq --help.
| Option | What it does |
|---|---|
-r, --raw-output | Print strings without JSON quotes. |
-j, --join-output | Like -r, and print no newline after each output. |
-c, --compact-output | Print each result on a single line. |
-n, --null-input | Do not read input; run the filter once with null as input. |
-s, --slurp | Read all inputs into one array and run the filter once. |
-e, --exit-status | Set the exit status from the last output (1 if false or null). |
-S, --sort-keys | Sort the keys of every object in the output. |
-R, --raw-input | Read each input line as a string instead of parsing JSON. |
-a, --ascii-output | Escape non-ASCII characters in the output. |
--tab | Indent output with tabs. |
--indent N | Indent output with N spaces. |
--arg NAME VALUE | Define $NAME as the string VALUE inside the filter. |
--argjson NAME JSON | Define $NAME as the parsed JSON value inside the filter. |
--slurpfile NAME FILE | Define $NAME as an array of the JSON values in FILE. |
--rawfile NAME FILE | Define $NAME as the contents of FILE as one string. |
-f, --from-file FILE | Read the filter from FILE. |
--seq | Use the application/json-seq format for input and output. |
Related
curlTransfer data to or from a URL; the standard tool for HTTP requests, API calls and downloads from the shell.awkScan text as records and fields, and run a small program on each line that matches a pattern.sedEdit a stream of text with a script of commands, most often to substitute text or delete lines.grepPrint the lines of files (or standard input) that match a pattern.- QuotingSingle quotes, double quotes, backslashes and $'...', what each protects, and the unquoted-variable bugs they prevent, each shown failing and then fixed.
- Command substitutionCapturing a command's output with $(...), what happens to trailing newlines and stderr, how to check the exit status, and the local trap that hides failures.