opscape

Bash scripting guide · part 24 of 32

Process substitution

Use <(cmd) and >(cmd) to hand a command's output to a program that wants a file name, keep while-read loops in the current shell, and avoid its timing traps.

Some programs only accept file names: diff, comm, paste, join, anything that takes two inputs. A pipe gives a program just one standard input. Process substitution bridges the gap: bash runs a command in the background, connects its output to a pipe, and puts a file name for that pipe on the command line.

  • <(cmd) expands to a name you can read cmd's output from.
  • >(cmd) expands to a name you can write to, and what you write becomes cmd's input.

It is a bash, zsh and ksh feature, not POSIX; dash rejects it.

What the substitution turns into

See the file name bash substitutes

echo <(true) <(true)
ls -l <(true) | cut -d ' ' -f 1,9-
file -L <(echo hi)
output
/dev/fd/63 /dev/fd/62
lr-x------ /dev/fd/61 -> pipe:[849572609]
/dev/fd/61: fifo (named pipe)

Each <(...) became a path under /dev/fd/, a name for one of the shell's open file descriptors. The link points at an anonymous pipe, and file reports it as a FIFO. The program reading it just calls open() on a path, so it works with any tool that takes a file name.

Comparing the output of two commands

This is the use people meet first: diff and comm want two files, and creating temporary files just to compare two command outputs is tedious.

diff two command outputs without temp files

diff <(sort -u data/fruits.txt) <(printf '%s\n' apple banana cherry kiwi)
explain
outputexit 1
4,6c4
< date
< elderberry
< fig
---
> kiwi

The left side is the de-duplicated fruit list, the right side a list typed inline. diff reads both "files" and reports that lines 4 to 6 on the left (date, elderberry, fig) were replaced by kiwi. It exits 1 because the inputs differ, as diff always does.

Set operations on two lists with comm

comm -12 <(sort data/fruits.txt | uniq) <(printf '%s\n' cherry fig grape | sort)
echo "---"
comm -13 <(sort -u data/fruits.txt) <(printf '%s\n' cherry fig grape | sort)
output
cherry
fig
---
grape

comm needs sorted input, which process substitution provides on the fly. -12 hides the lines unique to either side and leaves the intersection (cherry, fig); -13 leaves the lines only in the second list (grape).

Glue together columns produced by different commands

paste -d ' ' <(cut -d ' ' -f 1 data/hosts.txt) <(cut -d ' ' -f 2 data/hosts.txt | tr . -)
explain
output
web01 10-0-0-11
web02 10-0-0-12
db01 10-0-0-21
cache01 10-0-0-31

Two pipelines, each producing one column, are read side by side by paste. The second column was transformed (tr . -) independently of the first.

Feeding a while loop without losing variables

cmd | while read ... runs the loop in a subshell, so variables set inside it vanish (see Pipes and pipelines). Redirecting the loop's input from a process substitution keeps the loop in the current shell. Note the space: < <(cmd) is a < redirection whose file is <(cmd).

Collect values from a loop

total=0
while read -r name qty price; do
  total=$((total + qty))
done < <(tail -n +2 data/inventory.tsv)
echo "total quantity: $total"
output
total quantity: 20

tail -n +2 skips the header row, and the loop sums the second column. Because the loop ran in the current shell, total keeps its value after done. The same applies to mapfile -t lines < <(cmd).

Writing to a process: >(cmd)

The output form is most useful with tee, which writes to every file name it is given. Each >(...) receives a full copy of the stream, so one pass over the data can feed several consumers.

Send one stream to several commands

tee >(wc -l | sed 's/^/lines: /') >(awk '{ s += $1 } END { print "sum: " s }') \
  > /dev/null < data/numbers.txt | cat
output
sum: 324
lines: 10

tee copied the numbers into both substitutions: one counted the lines, the other added them up. tee's own copy went to /dev/null. The two results can arrive in either order, because the substitutions run in parallel. The trailing | cat is not decoration; the timing trap below explains why it is there.

The timing trap

Bash does not wait for process substitutions. The main command can finish, and the next line of the script can run, while the substituted process is still working, or has not even seen the end of its input yet.

Reading the result too early

seq 5 | tee >(sleep 0.2; wc -l > count.txt) > /dev/null
cat count.txt
echo "--- a moment later:"
sleep 0.5
cat count.txt
output
cat: count.txt: No such file or directory
--- a moment later:
5

tee finished first, and cat ran before the background wc had written anything, so the file did not exist yet. Half a second later it does. Sleeping is not a fix, it only makes the race rarer.

The reliable way to wait is to make something read the substitutions' output to the end. In tee >(...) >(...) | cat above, the substitutions inherit the pipe to cat as their standard output, and cat sees end of file only when every one of them has exited, so the pipeline cannot finish early. Bash also sets $! to the PID of the last process substitution so that wait $! can wait for it, but in our tests with bash 5.2 that hung when the command ran at the top level of a bash -c script, because the shell kept the write ends of the pipes open. When the result has to go into files, a temporary file and a plain sequence of commands is usually the clearer design.

Errors in a substitution are not reported

The exit status of <(cmd) is not part of the command's status, and set -e and pipefail do not see it either.

A failing substitution goes unnoticed

set -euo pipefail
sort <(grep ERROR missing.log)
echo "still running, status $?"
output
grep: missing.log: No such file or directory
still running, status 0

grep failed (the message is visible), yet sort happily sorted an empty input and returned 0, and the strict-mode script carried on. If the data matters, run the command separately and check it, or capture its output first: out=$(grep ERROR file) does fail under set -e.

Programs that need a real file

A process substitution is a pipe: it can be read once, front to back. A program that seeks in its input, reads it twice, or checks its size first will misbehave. Examples include some archive tools and editors. If a tool behaves oddly with <(...), use a temporary file from mktemp instead (see Bash idioms for real scripts).

Not available in sh

diff <(sort todo.txt) todo.txt
explain
outputexit 2
/bin/sh: 1: Syntax error: "(" unexpected

Under dash, ( after < is a syntax error, so #!/bin/sh scripts must use temporary files or named pipes from mkfifo instead. See POSIX sh vs bash.

Related