opscape

Bash scripting guide · part 5 of 32

Quoting

Single quotes, double quotes, backslashes and $'...', what each protects, and the unquoted-variable bugs they prevent, each shown failing and then fixed.

Most shell bugs are quoting bugs. The rule that prevents nearly all of them fits on one line: put double quotes around every variable expansion and command substitution, "$var" and "$(cmd)", unless you can say exactly why this one must stay unquoted.

The reason is what bash does to an unquoted $var after substituting its value. It splits the result into separate words on spaces, tabs and newlines (Word splitting), and then treats each word containing *, ? or [ as a filename pattern (Globbing). Quotes turn both steps off. Then the value arrives as exactly one argument, byte for byte.

The bug: a filename with a space

The fixture has a file called file with spaces.txt. Store its name in a variable and use it unquoted:

An unquoted variable splits a filename

f="file with spaces.txt"
wc -c $f
outputexit 1
wc: file: No such file or directory
wc: with: No such file or directory
wc: spaces.txt: No such file or directory
0 total

$f was split into three words, so wc was asked for three files called file, with and spaces.txt, none of which exist. The quotes in the assignment did not help: they only affect how that line is parsed, and are not stored in the value.

Quote the expansion

f="file with spaces.txt"
wc -c "$f"
output
34 file with spaces.txt

With "$f", wc receives one argument and counts the file's 34 bytes. Nothing else changed.

The bug: a value that looks like a pattern

Splitting is only half of it. After splitting, each unquoted word is matched against filenames as a glob. A message containing * turns into a directory listing.

An unquoted * expands to filenames

msg="* deploy at 5pm"
echo $msg
echo "$msg"
output
README.md access.log data file with spaces.txt link-to-notes logs notes.txt photos scripts src todo.txt deploy at 5pm
* deploy at 5pm

Unquoted, the * became every non-hidden name in the current directory, sorted, followed by the rest of the message. Quoted, it is just a character. The same thing happens to a value like [a] or file?.txt, which match only when a suitable file exists, so the bug comes and goes depending on the directory you run in.

The bug: an empty variable disappears

An unquoted variable that is empty does not become an empty argument: it vanishes entirely. In a test, that leaves an operator with nothing on one side.

An empty variable breaks a test

answer=""
if [ $answer = yes ]; then echo "confirmed"; fi
if [ "$answer" = yes ]; then echo "confirmed"; else echo "not confirmed"; fi
output
/bin/bash: line 2: [: =: unary operator expected
not confirmed

The first test became [ = yes ], which [ cannot parse: unary operator expected. The if treats the error as false and moves on, which is worse than a crash. Quoted, the test becomes [ "" = yes ], a valid comparison that is simply false. Bash's [[ ]] does not split variables, but quoting there costs nothing (Test operators).

Single quotes: everything literal

Inside single quotes nothing is special: no $, no backslash escapes, no globbing. What you type is what the command receives. Use them for fixed strings, especially ones full of shell characters: awk programs, sed scripts, regular expressions, jq filters.

The one thing you cannot put inside single quotes is a single quote. End the quoted part, add an escaped quote, and start again: 'it'\''s' is three pieces glued together, 'it', \' and 's'. Or use the $'...' form below.

Single quotes keep everything literal

price=5
echo 'Total: $price * 2 \n done'
echo "Total: $price * 2 \n done"
echo 'it'\''s fine'
output
Total: $price * 2 \n done
Total: 5 * 2 \n done
it's fine

Single quotes printed $price and \n as typed. Double quotes expanded $price but left * alone and the \n too, since echo in bash does not interpret escapes without -e. The last line shows the '\'' idiom for an apostrophe.

Double quotes: variables yes, splitting no

Inside double quotes, $var, ${...}, $(...), `...` and $((...)) are expanded, but the results are not split or globbed. A backslash is only special before $, `, ", \ and a newline. (In an interactive shell, ! also triggers history expansion inside double quotes; scripts are not affected.) Everything else, including *, ~, ' and #, is literal.

Double quotes nest correctly through command substitution, because $( ) starts a fresh quoting context. "$(dirname "$path")" looks wrong and is exactly right.

Quotes inside $( ) inside quotes

path="reports/Q1 summary/final draft.pdf"
echo "dir: $(dirname "$path")"
echo "base: $(basename "$path" .pdf)"
echo "the tilde stays literal in quotes: ~ and \$HOME is $HOME"
output
dir: reports/Q1 summary
base: final draft
the tilde stays literal in quotes: ~ and $HOME is /home/dev

The inner quotes belong to the command substitution, not to the outer string, so dirname and basename each get one argument. ~ is only expanded at the start of an unquoted word, so inside quotes write $HOME. \$ prints a literal dollar sign.

$'...': escapes like C

Bash's ANSI-C quoting, $'...', works like single quotes but understands backslash escapes: \n, \t, \e, \x41, é and \'. It is the readable way to put a tab, a newline or a single quote into an argument. dash does not support it; use printf there.

Tabs, newlines and apostrophes with $'...'

printf '%s\n' $'col1\tcol2' $'two\nlines' $'it\'s here'
IFS=$'\t' read -r name qty price < <(sed -n 2p data/inventory.tsv)
echo "name=$name qty=$qty price=$price"
output
col1	col2
two
lines
it's here
name=widget qty=12 price=4.50

Each $'...' is one argument containing a real tab, newline or apostrophe. IFS=$'\t' is the common real-world use: splitting tab-separated data, here the first data row of data/inventory.tsv.

"$@" keeps your arguments intact

"$@" expands to all positional parameters, each one a separate quoted word. Unquoted $@ and $* re-split every argument. "$*" joins them all into a single string with the first character of IFS (a space by default). In a wrapper script that passes its arguments on to another command, "$@" is almost always what you want.

"$@" versus $@ versus "$*"

args.sh
#!/usr/bin/env bash
printf 'quoted @: [%s]\n' "$@"
printf 'bare @:   [%s]\n' $@
printf 'quoted *: [%s]\n' "$*"
bash args.sh "file with spaces.txt" notes.txt
explain
output
quoted @: [file with spaces.txt]
quoted @: [notes.txt]
bare @:   [file]
bare @:   [with]
bare @:   [spaces.txt]
bare @:   [notes.txt]
quoted *: [file with spaces.txt notes.txt]

Two arguments went in. "$@" passed exactly two along. Bare $@ split the first one and produced four. "$*" produced one string with the arguments joined by a space.

Quotes are syntax, not data

Quote characters are removed after the shell parses a line; they are never part of the value unless they were themselves quoted. That is why storing a command line in a string does not work: the quotes inside the string are ordinary characters by the time the variable is expanded.

Quotes stored in a variable do nothing

opts="-name '*.txt'"
find . -maxdepth 1 $opts
echo "find exited with status $?"
output
find exited with status 0

$opts split into -name and '*.txt', single quotes included, so find looked for names that start and end with a literal apostrophe, matched nothing and still exited 0. Quoting "$opts" would not help either: then find would receive one argument with a space in it.

Store arguments in an array instead

opts=(-name '*.txt')
find . -maxdepth 1 "${opts[@]}"
output
./notes.txt
./file with spaces.txt
./todo.txt

An array keeps each argument as a separate element, and "${opts[@]}" expands to exactly those elements, so find sees -name and *.txt. This is the general answer whenever you build a command line piece by piece (Arrays).

Passing a value to another shell

bash -c, ssh host '...' and su -c take a string that another shell parses again. Pasting a variable into that string with double quotes is a second round of parsing, and any quote or $ in the value breaks it, or worse, runs. Pass values as arguments instead and refer to them as "$1" in the inner script.

Pass data to bash -c as arguments

name="Ann; echo INJECTED"
bash -c "echo Hello $name"
bash -c 'echo "Hello $1"' _ "$name"
printf '%q\n' "$name"
output
Hello Ann
INJECTED
Hello Ann; echo INJECTED
Ann\;\ echo\ INJECTED

Interpolated into the string, the ; in the value ended the inner echo and the rest ran as a second command. A name like O'Brien would instead leave a quote unclosed and break the inner script. Passed as an argument (the _ fills $0), the value is just data. printf %q prints a value in a form that is safe to paste into shell code, which is the tool for the rare case where you must build a string.

When leaving quotes off is right

  • Assignments: a=$b and a=$(cmd) do not split or glob, so quotes are optional there. Bash treats export a=$b and local a=$b the same way, but some other shells (older dash among them) split those, so many people simply quote every assignment.
  • Inside [[ ]] and (( )), variables are not split. On the right side of == in [[ ]], leaving a variable unquoted is how you ask for pattern matching; quote it to compare literally.
  • When you want splitting, such as a list of words you control. Prefer an array even then.

Tools help: shellcheck flags almost every unquoted expansion with a reason. Run it on every script you write.

Related