opscape

Bash scripting guide · part 6 of 32

Word splitting

When bash splits expansion results into separate words, how IFS controls it, and why for line in $(cat file) does not loop over lines.

After bash substitutes variables, command substitutions and arithmetic into a command line, it takes the unquoted results and splits them into separate words. This step, word splitting, is what makes $f turn into three arguments when f holds file with spaces.txt (Quoting). Understanding exactly when it happens lets you use it on purpose, and turn it off everywhere else.

Three facts cover most of it:

  1. Only the results of unquoted $var, ${...}, $(...), `...` and $((...)) are split. Text you typed literally is split by the parser, which is a different, earlier step.
  2. The characters that split are the ones in the IFS variable, by default space, tab and newline.
  3. Double quotes prevent it completely.

The easiest way to watch splitting is printf '[%s]\n', which prints each argument it receives on its own line, in brackets.

See how many words an expansion becomes

v="  one   two
three  "
printf '[%s]\n' $v
echo ---
printf '[%s]\n' "$v"
output
[one]
[two]
[three]
---
[  one   two
three  ]

Unquoted, the value became three words. Leading and trailing whitespace was dropped and runs of spaces and the newline counted as one separator. Quoted, it is one argument with every space and the newline intact.

Literal text is not split again

Splitting applies to what an expansion produced, not to the command as you wrote it. A variable's value is never re-parsed for quotes, ;, | or $. Those were handled before expansion and are ordinary characters in a value. Only IFS characters, and afterwards glob characters, matter.

Quotes and semicolons in a value stay literal

v='a "b c" ; d'
printf '[%s]\n' $v
output
[a]
["b]
[c"]
[;]
[d]

The double quotes inside the value did not group b c, and the ; did not end the command: splitting cut only at spaces, giving "b, c" and ; as separate words.

The for line in $(cat file) bug

Command substitution output is split the same way, which is a problem when you meant to get lines. $(cat file) splits on spaces as well as newlines, so you get words.

Looping over $(cat file) gives words, not lines

for line in $(cat data/poem.txt); do
  echo "line: $line"
done | head -n 6
output
line: The
line: quick
line: brown
line: fox
line: jumps
line: over

The first line of the poem, The quick brown fox, arrived as four separate iterations. Output is cut to six lines with head. Unquoted results are also globbed, so a line containing * would expand to filenames.

Read lines with while read

while IFS= read -r line; do
  echo "line: $line"
done < data/poem.txt
output
line: The quick brown fox
line: jumps over the lazy dog.
line: The dog sleeps;
line: the fox runs on.

read takes one line per call, and IFS= stops it from trimming leading and trailing whitespace. This is the correct line loop; Reading files line by line explains each part.

Changing IFS

Setting IFS to other characters makes them the separators. There is an important asymmetry: whitespace characters in IFS (space, tab, newline) are trimmed at the ends and runs of them count as one separator, while every other character separates exactly once, so two in a row make an empty field between them.

Split a comma-separated value

row="2,Bob Okafor,,sales"
IFS=,
printf '[%s]\n' $row
output
[2]
[Bob Okafor]
[]
[sales]

With IFS=, the space in Bob Okafor no longer splits, and the two adjacent commas produce an empty field, so four fields come out as four words. Setting IFS globally like this affects every later unquoted expansion in the script, so in real code scope it.

Changing IFS for the whole shell is how subtle bugs start much later in a script. Scope it instead: IFS=, read -r ... applies to that one read, and inside a function local IFS=, restores the old value on return. To split a string into an array, read -ra with a one-command IFS is the clean form.

Split into an array without touching the global IFS

row="2,Bob Okafor,bob@example.com,sales"
IFS=, read -ra fields <<< "$row"
echo "${#fields[@]} fields; name is ${fields[1]}"
printf 'IFS is still space-tab-newline: %q\n' "$IFS"
output
4 fields; name is Bob Okafor
IFS is still space-tab-newline: $' \t\n'

The IFS=, prefix only applies to read. -a fields stores the words in an array, and -r keeps backslashes literal. Afterwards IFS is the default again; printf %q shows it as $' \t\n'.

Splitting on newlines only

To split command output into lines while keeping spaces inside each line, set IFS to just a newline. Globbing still applies to the results, so turn it off with set -f if lines may contain *, ? or [. In bash, mapfile (also called readarray) is simpler and avoids both issues.

Split output into lines, two ways

IFS=$'\n'
set -f
lines=( $(grep fix todo.txt) )
set +f; unset IFS
printf '[%s]\n' "${lines[@]}"
mapfile -t lines2 < <(grep fix todo.txt)
printf '<%s>\n' "${lines2[@]}"
output
[fix login bug]
[fix logout bug]
<fix login bug>
<fix logout bug>

Both produce two elements, one per matching line, with the spaces inside each line kept. unset IFS restores default splitting behavior. The mapfile version needs no global state at all.

Empty IFS and the "$*" join

With IFS='', no splitting happens at all (globbing still does). The first character of IFS also has a second job: it is the separator "$*" and "${array[*]}" use when they join elements into one string, which gives a neat join idiom.

Join array elements with a character

hosts=(web01 web02 db01)
(IFS=,; echo "${hosts[*]}")
echo "default join: ${hosts[*]}"
output
web01,web02,db01
default join: web01 web02 db01

In the subshell, IFS starts with a comma, so "${hosts[*]}" joins with commas. Running it in ( ) keeps the change from leaking. Outside, the default IFS joins with a space.

Where splitting does not happen

  • Inside double quotes, always.
  • On the right side of a plain assignment: a=$b and a=$(cmd) copy the value whole.
  • Inside [[ ]] and (( )).
  • In the word after case and in here-strings (<<< $v).

Everywhere else, assume an unquoted expansion will be split and globbed. When you need a list, use an array and "${arr[@]}"; when you need lines, use while read or mapfile.

Related