opscape

Bash scripting guide · part 10 of 32

Arrays

Indexed arrays in bash, creating, reading, appending, looping, slicing and removing elements, and using them to build command lines safely.

A bash array holds a list of strings, each one kept intact no matter what characters it contains. That makes arrays the right tool for lists of filenames, hosts or command-line arguments, the places where people otherwise reach for a space-separated string and get bitten by word splitting (Word splitting).

Bash has indexed arrays (this page) and associative arrays keyed by strings (Associative arrays). POSIX sh has neither; its only list is "$@".

Creating and reading

Create an array and read its elements

servers=(web01 web02 "db 01")
echo "first: ${servers[0]}"
echo "last: ${servers[-1]}"
echo "count: ${#servers[@]}"
printf '[%s]\n' "${servers[@]}"
output
first: web01
last: db 01
count: 3
[web01]
[web02]
[db 01]

Indexes start at 0, and negative indexes count from the end (bash 4.3 and later). ${#servers[@]} is the number of elements. "${servers[@]}" expands to every element as a separate word, so db 01 stays one element despite its space.

The braces are required. $servers[1] is $servers (which means element 0) followed by the literal text [1]. And always write "${arr[@]}" with the quotes: without them, every element is split and globbed again, undoing the point of the array.

Two classic mistakes

servers=(web01 web02 "db 01")
echo "$servers[1]"
printf '[%s]\n' ${servers[@]}
output
web01[1]
[web01]
[web02]
[db]
[01]

$servers[1] printed element 0 followed by [1]. The unquoted ${servers[@]} split db 01 into two words, so four arguments came out of a three-element array.

Adding, changing and removing

arr+=(x y) appends elements. arr[i]=x sets one element. unset 'arr[i]' removes one, but leaves a gap: bash arrays are sparse, and indexes are not renumbered. ${!arr[@]} lists the indexes that actually exist. (Quote the unset argument, because arr[i] is also a valid glob pattern.)

Append, replace and remove elements

queue=(build test)
queue+=(package deploy)
queue[1]=lint
unset 'queue[2]'
declare -p queue
echo "indexes: ${!queue[@]}, count: ${#queue[@]}"
queue=("${queue[@]}")
declare -p queue
output
declare -a queue=([0]="build" [1]="lint" [3]="deploy")
indexes: 0 1 3, count: 3
declare -a queue=([0]="build" [1]="lint" [2]="deploy")

After the unset, index 2 is simply missing and the count is 3, while the highest index is still 3. Reassigning the array from "${queue[@]}" packs the elements back to 0, 1, 2. declare -p is the clearest way to see an array's real contents.

Looping

Loop over the values with for x in "${arr[@]}". When you also need positions, loop over "${!arr[@]}", which works correctly for sparse arrays too.

Loop over values and over indexes

files=(notes.txt todo.txt "file with spaces.txt")
for f in "${files[@]}"; do
  printf '%-22s %s lines\n' "$f" "$(wc -l < "$f")"
done
for i in "${!files[@]}"; do
  echo "$i: ${files[i]}"
done
output
notes.txt              6 lines
todo.txt               5 lines
file with spaces.txt   1 lines
0: notes.txt
1: todo.txt
2: file with spaces.txt

The file with spaces in its name works in both loops because each element stays one word. Inside [ ], the index is an arithmetic expression, so ${files[i]} needs no $ on i.

Filling an array from files or output

A glob fills an array with matching filenames, safely. For command output, mapfile -t (also spelled readarray) reads one element per line. Avoid arr=( $(cmd) ), which splits lines on spaces and expands any * in the output.

Arrays from a glob and from command output

logs=(logs/*.log)
echo "${#logs[@]} log files: ${logs[*]}"
mapfile -t hosts < <(cut -d' ' -f1 data/hosts.txt)
echo "${#hosts[@]} hosts, the last is ${hosts[-1]}"
shopt -s nullglob
none=(*.xyz)
echo "no matches gives ${#none[@]} elements"
output
2 log files: logs/app.log logs/error.log
4 hosts, the last is cache01
no matches gives 0 elements

The glob produced two elements, sorted. mapfile -t read four lines from cut, with -t removing each trailing newline. Without nullglob, a glob with no matches would leave the literal *.xyz as a single element; with it, the array is empty.

Slicing, joining, testing for an element

"${arr[@]:start:count}" takes a slice of elements. "${arr[*]}" joins all elements into one string with the first character of IFS. There is no built-in "contains" test; loop, or join and pattern-match when the values have no spaces.

Slice and join

steps=(fetch build test package deploy)
echo "middle: ${steps[@]:1:3}"
echo "from index 3: ${steps[@]:3}"
(IFS=,; echo "csv: ${steps[*]}")
printf -v joined '%s > ' "${steps[@]}"
echo "pipeline: ${joined% > }"
want=test
for s in "${steps[@]}"; do
  if [[ $s == "$want" ]]; then echo "contains $want"; break; fi
done
output
middle: build test package
from index 3: package deploy
csv: fetch,build,test,package,deploy
pipeline: fetch > build > test > package > deploy
contains test

Slices count elements, not characters. "${steps[*]}" joins with the first character of IFS only, so it suits single-character separators. For a longer one, printf -v repeats the format for every element into a variable, and ${joined% > } trims the extra separator at the end. The break stops the search at the first match.

Building a command line

The most practical use of arrays in scripts: assemble arguments conditionally, then run the command with "${args[@]}". Every argument arrives exactly as added, spaces and all, with no eval needed.

Assemble arguments conditionally

verbose=1
pattern="fix log"
args=(-n)
(( verbose )) && args+=(--color=never -i)
args+=(-e "$pattern" --)
grep "${args[@]}" todo.txt notes.txt
declare -p args
output
todo.txt:2:fix login bug
todo.txt:4:fix logout bug
declare -a args=([0]="-n" [1]="--color=never" [2]="-i" [3]="-e" [4]="fix log" [5]="--")

grep received the pattern fix log as one argument, and the -i only because verbose was set. The -- ends options before the filenames. The same approach works for rsync, curl or find invocations that grow with options.

Passing arrays to functions and limits

A function receives its arguments as "$@", so pass an array as fn "${arr[@]}". To pass one by name and modify it, use a nameref: local -n ref=$1 (bash 4.3). Arrays cannot be exported to child processes, and macOS's /bin/bash 3.2 lacks mapfile, negative indexes and namerefs.

Pass an array to a function

count_args() { echo "$# arguments, the last is '${!#}'"; }
add_item() { local -n list=$1; list+=("$2"); }
items=(one "two words")
count_args "${items[@]}"
add_item items "three"
declare -p items
output
2 arguments, the last is 'two words'
declare -a items=([0]="one" [1]="two words" [2]="three")

The function saw two arguments; ${!#} is indirection on $#, giving the last one. add_item changed the caller's array through the nameref list, which is an alias for items.

Related