Text processing
uniq
Collapse or report repeated adjacent lines; with sort, count how often each line occurs.
uniq [OPTION...] [INPUT [OUTPUT]]uniq removes adjacent duplicate lines: when the same line appears several times in a
row, it is printed once. Lines that repeat further apart are not touched, which is why uniq
is almost always used after sort, which brings equal lines together.
Its options turn it into a small reporting tool: -c prefixes each line with how many times it
occurred, -d prints only lines that were repeated, -u only lines that were not. The
pipeline sort | uniq -c | sort -rn (a frequency table, most common first) is one of the most
useful one-liners in the shell, for log analysis in particular.
uniq can compare only part of each line: -f N skips the first N fields, -s N skips N
characters and -w N compares at most N characters. It cannot pick a field in the middle;
for that, extract the field with cut or awk first, or use sort -u -k.
Examples
Only adjacent duplicates are removed
uniq data/fruits.txtbanana apple cherry apple date banana apple elderberry cherry fig
Nothing changes: no two equal lines are next to each other in fruits.txt, so uniq keeps
all ten. This is the most common surprise with uniq.
Remove all duplicates
sort data/fruits.txt | uniqapple banana cherry date elderberry fig
After sorting, equal lines are adjacent and uniq reduces each group to one line. sort -u does the same in one step.
Count occurrences
sort data/fruits.txt | uniq -c 3 apple
2 banana
2 cherry
1 date
1 elderberry
1 fig -c puts the size of each group in front of the line: apple occurs 3 times, date once.
Most frequent first
cut -d ' ' -f 3 logs/app.log | sort | uniq -c | sort -rn 6 INFO
4 WARN
2 ERROR The classic frequency table: extract the log level, group equal levels, count them, then sort
the counts numerically in reverse. The log has 6 INFO, 4 WARN and 2 ERROR lines.
Top clients in an access log
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -n 3 5 192.0.2.10
4 203.0.113.42
4 198.51.100.7 The same pattern on the client IP field answers "who is hitting the server most?"; head keeps the top three.
Show only the repeated lines
sort data/fruits.txt | uniq -dapple banana cherry
-d prints one copy of each line that occurs more than once: the fruits that were listed twice or more.
Show only the lines that are not repeated
sort data/fruits.txt | uniq -udate elderberry fig
-u is the opposite of -d: date, elderberry and fig occur exactly once.
Print every copy of the duplicates
sort data/fruits.txt | uniq -Dapple apple apple banana banana cherry cherry
-D keeps all lines that belong to a repeated group and drops the singles, useful to inspect what the duplicates look like.
Separate groups with a blank line
sort data/fruits.txt | uniq --group | head -n 8apple apple apple banana banana cherry
--group prints every line and puts an empty line between groups of equal lines.
Ignore case when comparing
printf 'Apple\napple\nAPPLE\nbanana\n' | uniq -i -c 3 Apple
1 banana -i treats the three spellings as one group; the first spelling seen, Apple, is the one printed.
Compare from a later field on
printf '08:01 disk full\n08:02 disk full\n08:05 backup done\n' | uniq -f 1 -c 2 08:01 disk full
1 08:05 backup done -f 1 skips the first field (the timestamp) when comparing, so two consecutive "disk full"
messages at different times count as repeats. The first line of each group is printed.
Compare only the first characters
printf 'web01 up\nweb02 up\ndb01 up\n' | uniq -w 3 -c 2 web01 up
1 db01 up -w 3 compares only the first three characters, so web01 and web02 form one group, and db01 another.
Trap: invisible differences break grouping
printf 'error \nerror\n' | uniq -c | cat -A 1 error $
1 error$ The first line has a trailing space, so the lines differ and both are counted once. cat -A
reveals it (error $). Trailing blanks and CRLF line endings are the usual culprits.
Common pitfalls
uniqonly collapses adjacent equal lines. Sort first, or usesort -u, orawk '!seen[$0]++'to remove duplicates while keeping the original order.sortanduniqmust agree on what equal means: sorting with-fand then runninguniqwithout-ileavesAppleandappleas separate lines.The second file argument is an output file:
uniq a.txt b.txtoverwritesb.txt. It does not compare two files; usecommordifffor that.uniq -cpads counts with leading spaces. To use them in a script, split withawkrather thancut -d ' '.
GNU vs BSD and macOS
The examples above ran the GNU/Linux version. These are the differences that matter on macOS and the BSDs (written from their documentation, not run here).
-D,--group,-wand-zare GNU extensions. BSD/macOSuniqhas-c,-d,-u,-i,-fand-s.
Options worth knowing
A selection, described in our own words. For the full list, see uniq(1) or uniq --help.
| Option | What it does |
|---|---|
-c, --count | Prefix each line with the number of times it occurred. |
-d, --repeated | Print only lines that are repeated, one copy each. |
-u, --unique | Print only lines that are not repeated. |
-D | Print all copies of repeated lines. |
--all-repeated[=METHOD] | Like -D, optionally separating groups: none, prepend or separate. |
--group[=METHOD] | Print all lines, separating groups with an empty line: separate, prepend, append or both. |
-i, --ignore-case | Ignore differences in case when comparing. |
-f, --skip-fields N | Skip the first N fields before comparing. |
-s, --skip-chars N | Skip the first N characters before comparing. |
-w, --check-chars N | Compare no more than N characters. |
-z, --zero-terminated | Lines end with NUL instead of newline. |
Related
sortSort lines of text alphabetically, numerically, by version or by chosen fields; also merge and de-duplicate.commCompare two sorted files line by line and show lines unique to each and lines in both.cutPrint selected fields, characters or bytes from each line, such as columns of a CSV or TSV file.awkScan text as records and fields, and run a small program on each line that matches a pattern.wcCount lines, words, characters and bytes in files or standard input.- Pipes and pipelinesHow cmd1 | cmd2 connects processes, which exit status a pipeline returns, PIPESTATUS, pipefail, SIGPIPE, and why variables set in a piped loop vanish.