Reading files line by line

The standard way to process a file line by line in bash is this loop:

while IFS= read -r line; do
  ...
done < file

Every piece of it prevents a specific bug. This page takes it apart, then covers the problems people hit next: a last line that disappears, counters that reset to zero, and commands inside the loop that swallow the input.

Why IFS= and -r

read splits the line using IFS and strips leading and trailing whitespace while doing it, and it treats backslashes as escape characters, joining a line ending in \ with the next one. For reading data exactly as it is in the file, you want neither. IFS= (empty, for this one command) disables the trimming; -r keeps backslashes literal.

$ printf '   indented line\npath C:\\new\\temp\n' > input.txt
> echo "--- plain read:"
> while read line; do echo "[$line]"; done < input.txt
> echo "--- IFS= read -r:"
> while IFS= read -r line; do echo "[$line]"; done < input.txt

--- plain read:
[indented line]
[path C:newtemp]
--- IFS= read -r:
[   indented line]
[path C:\new\temp]

Redirect into the loop, after done

The < file goes after done, so the whole loop reads from one open file and each read takes the next line. Writing read line < file inside the loop would reopen the file each time and read the first line forever.

$ n=0
> while IFS= read -r line; do
>   (( ++n ))
>   printf '%2d  %s\n' "$n" "$line"
> done < todo.txt
> echo "$n lines"

 1  buy milk
 2  fix login bug
 3  write tests
 4  fix logout bug
 5  deploy v2.4
5 lines

The missing last line

read returns failure when it reaches end of file before a newline, even if it read some characters. A file whose last line has no trailing newline, common with files written by editors on Windows or generated by programs, therefore loses that line: the data is in $line, but the loop body never runs for it.

$ printf 'alpha\nbeta\ngamma' > list.txt
> while IFS= read -r line; do echo "got: $line"; done < list.txt
> echo "but \$line still holds: $line"

got: alpha
got: beta
but $line still holds: gamma

$ printf 'alpha\nbeta\ngamma' > list.txt
> while IFS= read -r line || [[ -n $line ]]; do
>   echo "got: $line"
> done < list.txt

got: alpha
got: beta
got: gamma

The pipe-subshell trap

Each part of a pipeline runs in its own subshell. If the loop is on the right side of a |, it runs in a child process, and every variable it sets disappears when the pipeline ends. This is one of the most reported bash "bugs".

$ count=0
> grep fix todo.txt | while IFS= read -r line; do
>   (( ++count ))
>   echo "inside: $count $line"
> done
> echo "after the loop: count=$count"

inside: 1 fix login bug
inside: 2 fix logout bug
after the loop: count=0

There are three fixes. Redirect a file directly when there is one. For command output, use process substitution, < <(command), which feeds the loop through a file descriptor instead of a pipe, so the loop stays in the current shell (Process substitution). Or, in a script, set shopt -s lastpipe, which runs the last part of a pipeline in the current shell when job control is off, as it is in scripts.

$ count=0
> while IFS= read -r line; do
>   (( ++count ))
> done < <(grep fix todo.txt)
> echo "process substitution: count=$count"
> shopt -s lastpipe
> total=0
> grep fix todo.txt | while IFS= read -r line; do (( ++total )); done
> echo "lastpipe: total=$total"

process substitution: count=2
lastpipe: total=2

Splitting lines into fields

Give read several variable names and it splits the line by IFS; the last variable gets the rest of the line. Set IFS for the read alone, so it does not leak into the rest of the script. Use _ as a throwaway name for fields you skip.

$ {
>   read -r header
>   while IFS=, read -r id name email dept _; do
>     [[ $dept == engineering ]] && printf '%-14s %s\n' "$name" "$email"
>   done
> } < data/users.csv
> echo "header was: $header"

Alice Moreau   alice@example.com
Chen Wei       chen@example.com
Fatima Haddad  fatima@example.com
header was: id,name,email,dept,joined

mapfile: the whole file into an array

When you need all the lines at once, mapfile -t arr < file (also called readarray) reads each line into an array element, -t removing the newline. It is faster than a read loop and handles a missing final newline correctly. -s N skips N lines and -n N reads at most N.

$ mapfile -t lines < data/poem.txt
> echo "${#lines[@]} lines; the last is: ${lines[-1]}"
> mapfile -t -s 1 -n 3 rows < data/sales.csv
> printf 'row: %s\n' "${rows[@]}"
> printf 'alpha\nbeta\ngamma' | { mapfile -t m; echo "no final newline: ${#m[@]} elements"; }

4 lines; the last is: the fox runs on.
row: 2026-03-01,north,widget,12,4.50
row: 2026-03-01,south,gadget,3,19.99
row: 2026-03-02,east,widget,7,4.50
no final newline: 3 elements

Commands that steal the loop's input

Everything inside a while read loop shares its standard input. A command in the body that reads stdin, such as ssh, ffmpeg, another read, or a cat without arguments, consumes the rest of the file, and the loop ends after one iteration. Give the loop its own file descriptor with read -u 3 and 3< file, or give the inner command < /dev/null (ssh -n does that).

$ while read -r host ip; do
>   echo "checking $host"
>   ssh server 'echo "  remote saw: $(cat | wc -l) lines of stdin"'
> done < data/hosts.txt
> echo "--- with a separate descriptor:"
> while read -r -u 3 host ip; do
>   echo "checking $host ($ip)"
> done 3< data/hosts.txt

checking web01
  remote saw: 3 lines of stdin
--- with a separate descriptor:
checking web01 (10.0.0.11)
checking web02 (10.0.0.12)
checking db01 (10.0.0.21)
checking cache01 (10.0.0.31)

Windows line endings

A file with CRLF endings, like the fixture's data/windows.txt, leaves a carriage return at the end of every line read returns. It is invisible when printed, but breaks comparisons and shows up when the value is followed by more text. Strip it in the loop with ${line%$'\r'}, or convert the file first.

$ while IFS= read -r line; do
>   [[ $line == "line one" ]] && echo "matched as-is"
>   clean=${line%$'\r'}
>   [[ $clean == "line one" ]] && echo "matched after stripping \\r"
>   printf '%q -> %q\n' "$line" "$clean"
> done < data/windows.txt

matched after stripping \r
$'line one\r' -> line\ one
$'line two\r' -> line\ two