Pipes and pipelines: bash scripting guide

A pipe connects the standard output (fd 1) of one command to the standard input (fd 0) of the next. The shell creates the pipe, starts every command of the pipeline at the same time, each in its own process, and waits for all of them. Data flows through a small kernel buffer (64 KiB on Linux): the writer blocks when it is full, the reader blocks when it is empty, and nothing is ever written to disk.

Building a pipeline

Chain filters to answer a question

$ cut -d ' ' -f 9 access.log | sort | uniq -c | sort -rn

explain

output

Field 9 of the access log is the HTTP status. sort groups equal codes together, uniq -c counts each group, and sort -rn puts the most frequent first. Each tool does one small job and none of them knows about the others.

Only standard output goes through the pipe

Error messages are written to fd 2, which still points at the terminal. That is usually what you want: errors stay visible instead of being fed to the next program as data.

Errors skip the pipe

$ ls todo.txt missing.txt | tr a-z A-Z
> echo "--- with |&"
> ls todo.txt missing.txt |& tr a-z A-Z

output

ls: cannot access 'missing.txt': No such file or directory
TODO.TXT
--- with |&
LS: CANNOT ACCESS 'MISSING.TXT': NO SUCH FILE OR DIRECTORY
TODO.TXT

In the first pipeline, tr upper-cased the file name but the error reached the screen untouched. |& is bash shorthand for 2>&1 |: both streams go through the pipe, so the error came out upper-cased too. In sh scripts, write 2>&1 |.

The exit status of a pipeline

By default a pipeline's status is the status of its last command. A failure earlier in the chain is invisible to if, && and set -e. Bash records every status in the array PIPESTATUS, which is overwritten by the next command, so copy it immediately if you need it.

An early failure is hidden

$ grep ERROR missing.log | sort
> echo "pipeline status: $?"
> grep ERROR missing.log | sort
> echo "PIPESTATUS: ${PIPESTATUS[*]}"

output

grep: missing.log: No such file or directory
pipeline status: 0
grep: missing.log: No such file or directory
PIPESTATUS: 2 0

grep failed with status 2 because the file does not exist, but sort happily sorted nothing and returned 0, so $? is 0. PIPESTATUS keeps one status per command: 2 0.

Make the pipeline fail when any part fails

$ set -o pipefail
> grep ERROR missing.log | sort
> echo "pipeline status: $?"
> grep ERROR logs/app.log | wc -l
> echo "pipeline status: $?"

output

grep: missing.log: No such file or directory
pipeline status: 2
2
pipeline status: 0

With pipefail, the pipeline returns the status of the rightmost command that failed, here grep's 2. When everything succeeds it is still 0. This is why set -o pipefail is part of the usual strict-mode header; see Strict mode and its caveats for the catch that comes next.

SIGPIPE: when the reader stops early

If the reading end of a pipe is closed while the writer is still writing, the kernel sends the writer SIGPIPE, which terminates it. That is a feature: yes | head -n 2 would run forever otherwise. A process killed by a signal has status 128 + the signal number, and SIGPIPE is 13, so the writer reports 141.

head closes the pipe and the writer gets 141

$ yes | head -n 2
> echo "PIPESTATUS: ${PIPESTATUS[*]}"
> set -o pipefail
> seq 1000000 | head -n 1 > /dev/null
> echo "with pipefail: $?"

output

y
y
PIPESTATUS: 141 0
with pipefail: 141

yes was killed by SIGPIPE (141) after head printed two lines and exited. Normally nobody notices, but with pipefail that 141 becomes the pipeline's status, and under set -e the script dies on a perfectly good ... | head. A small input can escape this if the writer finishes before head exits, so the failure is intermittent, which makes it nastier.

Readers that drain the pipe avoid the 141

$ set -o pipefail
> seq 1000000 | sed -n 1p
> echo "sed -n 1p: $?"
> { seq 1000000 || [ $? -eq 141 ]; } | head -n 1
> echo "141 tolerated: $?"

output

1
sed -n 1p: 0
1
141 tolerated: 0

sed -n 1p prints the first line but keeps reading to the end, so seq is never cut off. That costs time on a huge input. The second form keeps the fast head and treats 141 from seq as success, so pipefail still catches real failures.

Workarounds: limit the output at the source (grep -m 1, seq 5), use a reader that consumes all its input instead of exiting early (sed -n 1p and awk 'NR==1' read to the end, unlike head or sed 1q), or accept 141 explicitly for that one producer.

Every part of a pipeline is a subshell

Because the commands run in parallel, bash forks a child process for each of them. A while read loop at the end of a pipeline therefore runs in a subshell, and any variable it sets disappears when the pipeline ends. This is the single most reported "bash bug" that is not a bug.

A counter set inside a piped loop is lost

$ count=0
> grep WARN logs/app.log | while read -r line; do
>   count=$((count + 1))
> done
> echo "warnings: $count"

output

warnings: 0

The loop did count four lines, but in a child process. The parent's count was never touched, so it prints 0.

Three ways to keep the variable

$ count=0
> while read -r line; do count=$((count + 1)); done < <(grep WARN logs/app.log)
> echo "process substitution: $count"
> 
> shopt -s lastpipe
> count=0
> grep WARN logs/app.log | while read -r line; do count=$((count + 1)); done
> echo "lastpipe: $count"
> 
> echo "or just ask grep: $(grep -c WARN logs/app.log)"

output

process substitution: 4
lastpipe: 4
or just ask grep: 4

Redirecting the loop's input from < <(...) keeps the loop in the current shell (see Process substitution). shopt -s lastpipe runs the last command of a pipeline in the current shell; it only takes effect when job control is off, which is the case in scripts but not at an interactive prompt. And often the cleanest fix is no loop at all.

The commands really run at the same time

A pipeline is not a sequence

$ SECONDS=0
> sleep 2 | echo "printed immediately"
> echo "pipeline took ${SECONDS}s"

output

printed immediately
pipeline took 2s

echo does not wait for sleep; it ignores its input, prints and exits at once. The pipeline as a whole still waits for every member, so it takes the 2 seconds of the slowest one. Ordering in a pipeline comes only from data: a command waits when it reads input that has not been written yet.

Negating a pipeline

A leading ! inverts the pipeline's final status, which is handy in conditions: if ! grep -q needle file; then .... Note that ! exempts the pipeline from set -e.

Act when something is absent

$ if ! grep -q "deploy v3" todo.txt; then
>   echo "v3 deploy is not on the list yet"
> fi

output

v3 deploy is not on the list yet

grep -q exits 1 because there is no match, ! turns that into success, and the then branch runs.

Buffering

When their output is a pipe instead of a terminal, most programs switch from line buffering to block buffering and write in chunks of several kilobytes. With a slow, continuous producer such as tail -f app.log | grep ERROR | cut ..., results then appear late and in bursts. Ask the middle stages to flush per line: grep --line-buffered, sed -u, awk with fflush(), or wrap a program in stdbuf -oL (coreutils) if it uses C stdio.

Building a pipeline

Only standard output goes through the pipe

The exit status of a pipeline

SIGPIPE: when the reader stops early

Every part of a pipeline is a subshell

The commands really run at the same time

Negating a pipeline

Buffering

Related