Debugging bash scripts

Bash has no debugger in the default install, but it does not need much of one. The shell can check a script without running it, print every command it executes with the arguments it actually received, and call a function of yours whenever something fails. Together with a static linter, that covers nearly every bug a shell script has.

Check the syntax first: bash -n

bash -n parses a script and reports syntax errors without executing anything. It is safe to run on a script that deletes files, and cheap enough to put in a pre-commit hook.

#!/bin/bash
for f in *.txt; do
  if [ -f "$f" ]; then
    echo "$f"
done

$ bash -n broken.sh

broken.sh: line 5: syntax error near unexpected token `done'
broken.sh: line 5: `done'

bash -n only finds what the parser finds. A misspelled variable, a missing quote around $file, or [ $x == y ] in a sh script are all valid syntax. For those, install ShellCheck (apt install shellcheck, brew install shellcheck) and run shellcheck script.sh; it knows hundreds of common shell mistakes and explains each warning with a code like SC2086 that you can look up.

Tracing with set -x

set -x (xtrace) makes bash print each command after expansion, just before running it, prefixed with +. Because it shows the expanded words, it answers the most common debugging question directly: "what did that variable actually contain?"

#!/bin/bash
count() { local n; n=$(grep -c "$1" logs/app.log); echo "$1: $n"; }
for level in WARN ERROR; do
  count "$level"
done

$ bash -x report.sh

+ for level in WARN ERROR
+ count WARN
+ local n
++ grep -c WARN logs/app.log
+ n=4
+ echo 'WARN: 4'
WARN: 4
+ for level in WARN ERROR
+ count ERROR
+ local n
++ grep -c ERROR logs/app.log
+ n=2
+ echo 'ERROR: 2'
ERROR: 2

$ name="file with spaces.txt"
> set -x
> wc -l $name
> wc -l "$name"
> set +x
> echo "tracing is off again"

+ wc -l file with spaces.txt
wc: file: No such file or directory
wc: with: No such file or directory
wc: spaces.txt: No such file or directory
0 total
+ wc -l 'file with spaces.txt'
1 file with spaces.txt
+ set +x
tracing is off again

Line numbers in the trace: PS4

The + prefix is the value of the PS4 variable, and bash expands it before printing each trace line. Putting $LINENO, $BASH_SOURCE and $FUNCNAME in it turns a trace into something you can map back to the source. Use single quotes so the variables are expanded at trace time, not when you assign PS4. The first character is repeated to show nesting depth, so keep it a +.

#!/bin/bash
PS4='+ ${BASH_SOURCE}:${LINENO}:${FUNCNAME[0]:+${FUNCNAME[0]}():} '
set -x
target=${1:-staging}
check() {
  test -f "$1"
}
check data/config.json
echo "deploying to $target"

$ bash deploy.sh

+ deploy.sh:4: target=staging
+ deploy.sh:8: check data/config.json
+ deploy.sh:6:check(): test -f data/config.json
+ deploy.sh:9: echo 'deploying to staging'
deploying to staging

Sending the trace somewhere else: BASH_XTRACEFD

Tracing to standard error is a nuisance when the script's own error messages are what you want to read, or when stderr is captured by something else. Set BASH_XTRACEFD to a file descriptor number and bash writes the trace there instead.

$ exec 5> trace.log
> BASH_XTRACEFD=5
> set -x
> total=0
> for n in 3 4; do total=$((total + n)); done
> echo "total is $total"
> set +x
> echo "--- trace.log:"
> cat trace.log

total is 7
--- trace.log:
+ total=0
+ for n in 3 4
+ total=3
+ for n in 3 4
+ total=7
+ echo 'total is 7'
+ set +x

Catching the failure: trap ERR

An ERR trap runs whenever a command returns a non-zero status, under the same rules that decide when set -e would exit (not in if conditions, not on the left of && or ||). Inside the trap, $LINENO is the line of the failing command and $BASH_COMMAND is its text. $? is its status, but only on the first line of the trap, before anything else overwrites it.

#!/bin/bash
set -eE
trap 'echo "error $? at line $LINENO: $BASH_COMMAND" >&2' ERR
load() {
  grep -q "^port" data/settings.conf
  grep -q "^workers" data/settings.conf
}
echo "loading"
load
echo "never reached"

$ bash job.sh

loading
error 1 at line 6: grep -q "^workers" data/settings.conf

A stack trace with caller

For deeper scripts the line alone is not always enough. The builtin caller N reports the line, function and file of the Nth frame of the call stack, and looping over it gives a backtrace.

#!/bin/bash
set -eE
backtrace() {
  local i=0
  echo "failed: $BASH_COMMAND" >&2
  while caller "$i" >&2; do i=$((i + 1)); done
}
trap backtrace ERR
inner() { false; }
outer() { inner; }
outer

$ bash stack.sh

failed: false
9 inner stack.sh
10 outer stack.sh
11 main stack.sh

Other tools worth knowing

• set -v prints each line as it is read, before expansion; occasionally useful to see a heredoc or a multi-line command exactly as written.
• declare -p var prints a variable with its type and exact contents, including arrays, which echo would flatten.
• printf '%q\n' "$var" shows invisible characters such as a trailing \r from a file with Windows line endings.

$ read -r first < data/windows.txt
> echo "[$first]"
> printf '%q\n' "$first"
> hosts=(web01 "db 01")
> declare -p hosts

[line one
]
$'line one\r'
declare -a hosts=([0]="web01" [1]="db 01")