Text processing
shuf
Shuffle lines into random order, or pick random lines, numbers or words.
shuf [OPTION...] [FILE] | shuf -e [ARG...] | shuf -i LO-HI
shuf prints its input lines in a random order. With -n N it stops after N lines, which makes
it a random sampler: one random line from a file, five random test cases, a random winner. It
can also shuffle its arguments (-e) or a range of integers (-i 1-100) without any input
file, and with -r it samples with replacement, so values can repeat, like dice rolls.
Every example on this page uses real randomness, so its output differs each time the page is
built; that is the point of the tool. When you need a repeatable shuffle (a fixed train/test
split, a reproducible test order), give shuf a fixed stream of bytes with --random-source:
the same bytes always produce the same order.
shuf is a GNU coreutils program. sort -R also shuffles, but it keeps identical lines next to
each other, so it is not a true shuffle.
Examples
Shuffle all lines
shuf todo.txt
buy milk fix login bug deploy v2.4 write tests fix logout bug
All five lines, in a random order that changes every run.
Pick random lines
shuf -n 2 todo.txt
write tests fix login bug
-n 2 outputs only two lines, picked at random without repeats.
A random line from a file
shuf -n 1 data/fruits.txt
apple
One line chosen uniformly at random. apple appears three times in the file, so it is three
times as likely as fig to be picked.
A random number in a range
shuf -i 1-6 -n 1
2
-i 1-6 shuffles the integers 1 to 6 instead of reading input; -n 1 keeps one, like rolling a die.
Shuffle words given on the command line
shuf -e red green blue yellow
blue green yellow red
-e treats each argument as an input line, so no file or pipe is needed.
Sample with replacement
shuf -r -n 10 -e H T | paste -s -d ' '
H H H T T T T H H H
-r lets values repeat, so this is ten independent coin tosses. Without -r, -n 10 could
not return more than the two values that exist.
A repeatable shuffle
shuf --random-source=<(yes 42) -i 1-10 | paste -s -d ' '
shuf --random-source=<(yes 42) -i 1-10 | paste -s -d ' '
3 7 5 10 1 6 9 8 2 4 3 7 5 10 1 6 9 8 2 4
--random-source reads "random" bytes from a file instead of the system's generator. Here
the endless output of yes 42 acts as a seed: both runs print the same order, and a
different seed string gives a different one. This is for reproducibility, not security.
Split data into random halves
tail -n +2 data/users.csv | shuf --random-source=<(yes split) > shuffled.csv
head -n 4 shuffled.csv > train.csv
tail -n +5 shuffled.csv > test.csv
wc -l train.csv test.csv
4 train.csv 4 test.csv 8 total
Shuffle once with a fixed seed, then cut the result: the 8 users are divided into two groups of 4, and rerunning gives the same groups.
Write the result to a file
shuf -n 3 -o picks.txt todo.txt
wc -l picks.txt
3 picks.txt
-o writes to a file, which may even be the input file, since shuf reads everything first.
Common pitfalls
sort -Ris not a shuffle: it sorts by a random hash of each line, so duplicate lines always end up adjacent. Useshufwhen duplicates must be spread out.Without
-n,shufholds the whole input in memory before printing anything. With-n, GNUshufcan sample a large input without keeping all of it.--random-sourcewith a short file fails with an end-of-file error once the bytes run out. Use an endless stream such as<(yes seed).shuf -iand-ecannot be combined with an input file.
GNU vs BSD and macOS
The examples above ran the GNU/Linux version. These are the differences that matter on macOS and the BSDs (written from their documentation, not run here).
macOS and the BSDs have no
shufin the base system. Install GNU coreutils (where it may be namedgshuf), or usesort -Ron macOS, with the duplicate caveat above.
Options worth knowing
A selection, described in our own words. For the full list, see shuf(1) or shuf --help.
| Option | What it does |
|---|---|
-n, --head-count COUNT | Output at most COUNT lines. |
-e, --echo | Treat each command-line argument as an input line. |
-i, --input-range LO-HI | Shuffle the integers from LO to HI. |
-r, --repeat | Allow output lines to repeat (sampling with replacement). |
-o, --output FILE | Write the result to FILE. |
--random-source FILE | Get random bytes from FILE, for repeatable results. |
-z, --zero-terminated | Lines end with NUL instead of newline. |