opscape

Bash scripting guide · part 15 of 32

Loops

for over lists, files and ranges, C-style for, while and until, break and continue, retry loops, and redirecting a whole loop's input or output.

Bash has three loop forms: for walks over a list of words, while repeats as long as a command succeeds, and until repeats as long as it fails. Each body sits between do and done. Like if, the loop conditions are commands and their exit statuses, not boolean expressions.

for over a list

for name in WORDS; do ...; done assigns each word to name in turn. The words go through the usual expansions, so they can come from literals, arrays, globs or brace expansion. Leaving out in WORDS loops over the script's arguments, "$@".

Loop over words, an array and the arguments

for env in dev staging prod; do echo "deploy to $env"; done
hosts=(web01 "web 02")
for h in "${hosts[@]}"; do echo "ping [$h]"; done
set -- one two
for arg; do echo "arg: $arg"; done
output
deploy to dev
deploy to staging
deploy to prod
ping [web01]
ping [web 02]
arg: one
arg: two

The array loop keeps web 02 as one item because "${hosts[@]}" is quoted. for arg with no in iterated over the positional parameters set by set --.

Looping over files

Use a glob directly in the for list. Each matching filename is one word, spaces included, and the names come out sorted. Never loop over $(ls), which splits names on spaces and mangles unusual characters.

Loop over files with a glob

for f in logs/*.log; do
  printf '%-16s %3d lines\n' "$f" "$(wc -l < "$f")"
done
for f in *.txt; do echo "text file: [$f]"; done
output
logs/app.log      12 lines
logs/error.log     2 lines
text file: [file with spaces.txt]
text file: [notes.txt]
text file: [todo.txt]

The glob handled file with spaces.txt correctly. Inside the loop, "$f" is quoted, as every variable holding a filename should be.

If nothing matches, bash leaves the pattern unchanged, and the loop runs once with the literal text *.bak as the "filename". Guard with [[ -e $f ]] || continue, or turn on nullglob so a non-matching glob expands to nothing.

A glob that matches nothing

for f in *.bak; do echo "would delete: $f"; done
shopt -s nullglob
for f in *.bak; do echo "would delete: $f"; done
echo "with nullglob the loop body never ran"
output
would delete: *.bak
with nullglob the loop body never ran

Without nullglob, the loop body ran once with the unexpanded pattern *.bak, which is how scripts end up running commands on a file called *.bak.

Ranges and C-style for

{1..5} and {01..10..3} are brace expansions: they generate words before anything else happens, which also means they cannot contain variables. For a range with variable bounds, use the arithmetic for (( init; condition; step )) loop, or seq.

Fixed ranges, variable ranges and C-style for

echo {1..5} / {01..10..3} / {a..e}
n=3
echo {1..$n}
for (( i = 1; i <= n; i++ )); do printf 'step %d of %d\n' "$i" "$n"; done
for i in $(seq 0 5 15); do printf '%s ' "$i"; done; echo
output
1 2 3 4 5 / 01 04 07 10 / a b c d e
{1..3}
step 1 of 3
step 2 of 3
step 3 of 3
0 5 10 15

Brace ranges support zero padding and a step. {1..$n} did not expand, because brace expansion runs before variables are substituted, so it printed literally. The C-style loop and seq both handle variable bounds.

while and until

while COMMAND; do ...; done runs the body as long as COMMAND exits 0. Any command works: (( )) for counters, [[ ]] for tests, read for input, pgrep for "while a process is running". until is the inverse and reads well for waiting.

Count with while, wait with until

n=3
while (( n > 0 )); do echo "T-minus $n"; (( n-- )); done
( sleep 1; touch ready.flag ) &
tries=0
until [[ -e ready.flag ]]; do
  (( ++tries )); sleep 0.2
done
echo "ready.flag appeared after $tries checks"
output
T-minus 3
T-minus 2
T-minus 1
ready.flag appeared after 5 checks

The until loop polled every 0.2 seconds until a background job created the file. The exact number of checks depends on timing. In real scripts, also cap the number of tries so a file that never appears does not hang the script.

break and continue

continue skips to the next iteration; break leaves the loop. Both take an optional number to act on an enclosing loop: break 2 exits two levels at once.

Skip and stop

while IFS= read -r line; do
  [[ $line == \#* || -z $line ]] && continue
  [[ $line == timeout* ]] && { echo "found: $line"; break; }
  echo "setting: $line"
done < data/settings.conf
for dir in data logs src; do
  for f in "$dir"/*; do
    [[ $f == *.log.1 ]] && { echo "old log in $dir, stopping both loops"; break 2; }
  done
  echo "$dir checked"
done
output
setting: host = 0.0.0.0
setting: port = 8080
setting: log_level = info
setting: log_file = /var/log/demo/app.log
found: timeout = 30
data checked
old log in logs, stopping both loops

The first loop skipped comments and blank lines with continue and stopped at timeout. In the second, break 2 left both the inner and the outer loop as soon as logs/app.log.1 turned up, so src was never checked and logs checked never printed.

Retry loops

A bounded retry is a for over attempts with a break on success. Put the command being retried in the if so a failure does not trip set -e.

Retry a command with a delay

ok=0
for attempt in 1 2 3; do
  if curl -fsS -o /dev/null "http://localhost:8080/status/503"; then
    ok=1; break
  fi
  echo "attempt $attempt failed"
  sleep "$attempt"
done
(( ok )) || echo "giving up after $attempt attempts"
output
curl: (22) The requested URL returned error: 503
attempt 1 failed
curl: (22) The requested URL returned error: 503
attempt 2 failed
curl: (22) The requested URL returned error: 503
attempt 3 failed
giving up after 3 attempts

The test server's /status/503 always fails, so curl -f returns an error each time; -sS hides the progress meter but still prints the error. The delay grows with each attempt, a simple backoff. After the loop, attempt keeps its last value and the ok flag says whether any attempt succeeded.

Redirecting a whole loop

A loop is a compound command, so you can redirect its input or output once, after done, instead of on every command inside. This is faster and keeps the file open across iterations. Piping into or out of a loop is also possible, but a pipe runs the loop in a subshell, so variables set inside are lost afterwards (Reading files line by line shows the fix).

Write and pipe a loop's output

for f in data/*.csv; do
  echo "$(head -n 1 "$f") <- $f"
done > headers.txt
cat headers.txt
for n in 5 3 9 1; do echo "$n"; done | sort -n | tr '\n' ' '; echo
output
date,region,product,units,price <- data/sales.csv
id,name,email,dept,joined <- data/users.csv
1 3 5 9

All output of the first loop went to headers.txt with a single redirection. The second loop's output was piped through sort as one stream.

Infinite loops

while true; do ...; done (or while :) runs until something inside breaks out or the script is killed. Combine it with sleep for polling, read for interactive menus, and always give it a clear exit: a break, an exit, or a Traps and signals handler for cleanup on Ctrl-C.

Related