opscape

Bash scripting guide · part 31 of 32

Bash idioms for real scripts

The small patterns that make scripts robust - die(), logging to stderr, ${1:?usage}, command -v checks, mktemp with trap cleanup, retries with backoff, locking and safe config parsing.

Most production shell scripts are built from the same dozen patterns. None of them is clever; each one exists because the naive version failed somebody at 3 a.m. This page collects them with a run of each, so you can see how they behave rather than take them on trust.

Messages go to stderr, data goes to stdout

A script's standard output is often piped into another program or captured with $(...). Progress messages, warnings and errors written there corrupt that data. Send them to fd 2, and give the script one exit point for fatal errors.

log() and die() helpers

count.sh
#!/bin/bash
log() { printf '%s: %s\n' "${0##*/}" "$*" >&2; }
die() { log "error: $*"; exit 1; }

[[ -r ${1-} ]] || die "cannot read '${1-}'"
log "counting lines in $1"
wc -l < "$1"
bash count.sh todo.txt
echo "--- captured: $(bash count.sh todo.txt 2>/dev/null)"
bash count.sh nope.txt
echo "exit status: $?"
output
count.sh: counting lines in todo.txt
5
--- captured: 5
count.sh: error: cannot read 'nope.txt'
exit status: 1

The log line appears on the terminal but is not part of the captured value: $(...) took only the 5 from stdout. die prints a prefixed message and exits 1, so the caller can test the status. Prefixing messages with the script's name (${0##*/}) tells you which script complained when several run in a pipeline.

Required arguments: ${1:?message}

${var:?message} expands the variable if it is set and non-empty; otherwise it prints the message to stderr and exits a non-interactive shell. It is a one-line usage check.

Abort with a usage message when an argument is missing

copy.sh
#!/bin/bash
src=${1:?usage: copy.sh SRC DEST}
dest=${2:?usage: copy.sh SRC DEST}
echo "would copy $src to $dest"
bash copy.sh todo.txt
echo "exit status: $?"
bash copy.sh todo.txt /tmp/todo.txt
output
copy.sh: line 3: 2: usage: copy.sh SRC DEST
exit status: 1
would copy todo.txt to /tmp/todo.txt

With one argument, the second expansion failed: bash printed the script name, line number, parameter name and message, and exited with status 1 before the echo. For a longer usage text, write a usage() function and test $# instead.

Check dependencies up front

command -v name prints how a command would be found and returns non-zero if it would not be. It is a builtin, POSIX, and works for functions and builtins as well as programs, unlike which, which is an external program whose behavior differs between systems.

Fail early if a required tool is missing

for tool in jq curl git; do
  if command -v "$tool" > /dev/null; then
    echo "found $tool at $(command -v "$tool")"
  else
    echo "missing: $tool" >&2
  fi
done
output
found jq at /usr/bin/jq
found curl at /usr/bin/curl
missing: git

jq and curl are installed in the sandbox, git is not. Checking everything at the start of a script and listing all the missing tools at once beats failing halfway through with a command not found after some work has been done.

Temporary files that clean up after themselves

mktemp creates a file with a unique, unpredictable name and mode 600, which avoids both collisions between concurrent runs and the classic /tmp/myscript.$$ symlink attack. An EXIT trap removes it however the script ends. See Traps and signals for the details of EXIT.

mktemp with an EXIT trap

tmp.sh
#!/bin/bash
set -euo pipefail
tmp=$(mktemp)
trap 'rm -f "$tmp"' EXIT
sort -u data/fruits.txt > "$tmp"
echo "working file: $tmp ($(wc -l < "$tmp") lines)"
false   # something fails
echo "not reached"
bash tmp.sh
echo "exit status: $?"
ls /tmp
output
working file: /tmp/tmp.RALER6vwMu (6 lines)
exit status: 1

The script failed on false, and set -e ended it with status 1, but the EXIT trap still removed the temporary file: /tmp is empty afterwards. Use mktemp -d for a whole scratch directory and rm -rf it in the trap.

Retrying with backoff

Network calls and lock contention fail transiently. A retry loop with an increasing delay handles that without hammering the other side. Keep the attempt count and delays explicit so the worst-case runtime is obvious.

Retry a command with exponential backoff

retry.sh
#!/bin/bash
retry() {
  local max=$1 delay=1 n=1
  shift
  until "$@"; do
    if (( n >= max )); then
      echo "giving up after $n attempts" >&2
      return 1
    fi
    echo "attempt $n failed, retrying in ${delay}s" >&2
    sleep "$delay"
    n=$((n + 1)) delay=$((delay * 2))
  done
}
retry 3 curl -fsS -o /dev/null http://localhost:8080/status/503
echo "retry returned $?"
retry 3 curl -fsS http://localhost:8080/api/status.json
bash retry.sh
explain
output
curl: (22) The requested URL returned error: 503
attempt 1 failed, retrying in 1s
curl: (22) The requested URL returned error: 503
attempt 2 failed, retrying in 2s
curl: (22) The requested URL returned error: 503
giving up after 3 attempts
retry returned 1
{"status": "ok", "version": "2.4.1", "uptime_s": 86400}

The first URL always answers 503, so curl -f fails three times with 1 and 2 seconds of waiting in between, then retry gives up and returns 1. The second succeeds on the first try. Passing the command as separate words ("$@") instead of a string avoids eval and keeps arguments with spaces intact. Real code often adds random jitter to the delay.

Only one copy at a time: locking

A script run from a scheduler can overlap with its previous run. flock (util-linux) holds a lock on a file descriptor for as long as the descriptor is open, and the kernel releases it when the process exits, even after kill -9. Where flock is not available, mkdir is the portable fallback, because creating a directory either succeeds or fails atomically.

Refuse to start if another run holds the lock

job.sh
#!/bin/bash
exec 9> /tmp/job.lock
if ! flock -n 9; then
  echo "another run is in progress, exiting" >&2
  exit 1
fi
echo "run $1 has the lock"
sleep 1
bash job.sh A &
sleep 0.3
bash job.sh B
echo "B exited with $?"
wait
bash job.sh C
output
run A has the lock
another run is in progress, exiting
B exited with 1
run C has the lock

Run A opened descriptor 9 on the lock file and took the lock. B tried with -n (non-blocking), failed immediately and exited 1. Once A finished, the lock was free and C got it. Without -n, flock waits; -w 10 waits at most 10 seconds.

The portable mkdir lock

lock=/tmp/myjob.lock
if mkdir "$lock" 2>/dev/null; then
  trap 'rmdir "$lock"' EXIT
  echo "lock acquired"
  mkdir "$lock" 2>/dev/null || echo "a second attempt fails while it is held"
fi
output
lock acquired
a second attempt fails while it is held

The first mkdir succeeded and installed a trap to remove the directory on exit; the second failed because the directory exists. The weakness of this method is a crash that skips the trap, such as kill -9: the directory stays and blocks every later run until someone removes it. flock has no such stale-lock problem.

Reading a config file without sourcing it

source config is tempting for key=value files, but it executes the file as code: any $(...) in it runs. When the file is not fully trusted, or uses key = value with spaces, parse it instead.

Parse key = value lines into an associative array

declare -A conf
while IFS='=' read -r key value; do
  key=${key//[[:space:]]/}
  [[ -z $key || $key == \#* ]] && continue
  value=${value#"${value%%[![:space:]]*}"}
  conf[$key]=$value
done < data/settings.conf
for k in host port log_level timeout; do echo "$k -> ${conf[$k]}"; done
echo "max_connections: ${conf[max_connections]:-unset}"
output
host -> 0.0.0.0
port -> 8080
log_level -> info
timeout -> 30
max_connections: unset

IFS='=' splits each line at the first = (read puts the rest of the line in the last variable). Spaces are stripped from the key and leading spaces from the value, and blank lines and # comments are skipped, including the commented-out max_connections. Nothing in the file is executed.

Defaults that callers can override

: "${VAR:=default}" assigns a default only when the variable is unset or empty. The : command does nothing; it exists so the expansion has something to be an argument of. This lets callers override settings through the environment without any option parsing.

Configurable through the environment

fetch.sh
#!/bin/bash
: "${API_URL:=http://localhost:8080}"
: "${TIMEOUT:=5}"
echo "GET $API_URL/api/status.json (timeout ${TIMEOUT}s)"
curl -sS --max-time "$TIMEOUT" "$API_URL/api/status.json"
bash fetch.sh
TIMEOUT=2 bash fetch.sh | grep GET
output
GET http://localhost:8080/api/status.json (timeout 5s)
{"status": "ok", "version": "2.4.1", "uptime_s": 86400}
GET http://localhost:8080/api/status.json (timeout 2s)

The first run used both defaults. The second set TIMEOUT only in the environment of that one command, and the script picked it up. The | grep GET keeps just the script's first line to show the changed value.

Putting it together

A skeleton that combines these idioms, as a starting point:

#!/usr/bin/env bash
set -euo pipefail

log() { printf '%s: %s\n' "${0##*/}" "$*" >&2; }
die() { log "error: $*"; exit 1; }

main() {
  local input=${1:?usage: ${0##*/} FILE}
  command -v jq > /dev/null || die "jq is required"
  tmp=$(mktemp -d)
  trap 'rm -rf "$tmp"' EXIT
  # ... work ...
}

main "$@"

Wrapping the body in main and calling it on the last line means bash has read the whole file before running anything, so editing the script while it runs cannot make it execute a half-written line. For the trade-offs of set -euo pipefail, read Strict mode and its caveats.

Related