Text processing
comm
Compare two sorted files line by line and show lines unique to each and lines in both.
comm [OPTION...] FILE1 FILE2
comm compares two sorted files and prints three columns: lines only in the first file,
lines only in the second (indented by one tab), and lines in both (indented by two). The options
-1, -2 and -3 hide the column with that number, so you combine them to keep the one you
want: comm -12 prints the intersection, comm -23 what is only in the first file, comm -13
what is only in the second.
That makes comm the set-operations tool of the shell: which hosts are in the inventory but
not responding, which packages were added, which users exist in one list and not the other.
Like join, it requires sorted input; sort both sides on the fly with <(sort file). To see
how two files differ in content and order, use diff instead.
Examples
The three columns
cache01
db01
web01
web02
build01
db01
web01
web02
comm expected.txt running.txt
build01 cache01 db01 web01 web02
cache01 is only in the first file (column 1), build01 only in the second (one tab in),
and the other three hosts are in both (two tabs in).
Lines in both files
cache01
db01
web01
web02
build01
db01
web01
web02
comm -12 expected.txt running.txt
db01 web01 web02
-12 hides columns 1 and 2, leaving only the common lines, with no indentation.
Lines only in the first file
cache01
db01
web01
web02
build01
db01
web01
web02
comm -23 expected.txt running.txt
cache01
-23 keeps column 1: hosts that should be running but are not. Here, cache01.
Lines only in the second file
cache01
db01
web01
web02
build01
db01
web01
web02
comm -13 expected.txt running.txt
build01
-13 keeps column 2: the unexpected host build01.
Compare unsorted lists
web02
web01
build01
db01
comm -3 <(sort inventory.txt) <(cut -d ' ' -f 1 data/hosts.txt | sort)
build01 cache01
Process substitution sorts both inputs without temporary files. -3 hides the common lines,
so only the differences remain: build01 is in the inventory but not in hosts.txt, and
cache01 (indented) is the other way round. No output would mean the lists match.
Everything that differs, from either side
comm -3 <(sort -u data/fruits.txt) <(printf '%s\n' apple banana kiwi | sort)
cherry date elderberry fig kiwi
With the common column hidden, lines unique to the first input are flush left and lines
unique to the second are indented by a tab: kiwi is new, the other fruits are missing.
Unsorted input is detected
comm data/fruits.txt <(sort -u data/fruits.txt)
apple banana comm: file 1 is not in sorted order apple cherry apple date banana apple elderberry cherry fig comm: input is not in sorted order
fruits.txt is not sorted. comm prints what it can, warns that file 1 is not in sorted
order and exits with 1. The output above the warning is not reliable.
Count each category
cache01
db01
web01
web02
build01
db01
web01
web02
comm --total -123 expected.txt running.txt
1 1 3 total
--total adds a summary line with the size of each column: 1 only in the first file, 1 only
in the second, 3 in both. Hiding all three columns leaves just that line.
Common pitfalls
Both inputs must be sorted in the current locale's order. Mixing
LC_ALL=C sortwith acommin another locale gives wrong results; use the same locale for both.Duplicate lines are compared one for one: a line twice in file 1 and once in file 2 shows up once as common and once as unique to file 1. Use
sort -uif you mean sets.Trailing spaces and CRLF line endings make lines that look identical compare as different. Clean them up with
sed 's/[[:space:]]*$//'ortr -d '\r'.
GNU vs BSD and macOS
The examples above ran the GNU/Linux version. These are the differences that matter on macOS and the BSDs (written from their documentation, not run here).
--total,--output-delimiter,--check-orderand--nocheck-orderare GNU options;-1,-2,-3and-iwork on BSD/macOS too.
Options worth knowing
A selection, described in our own words. For the full list, see comm(1) or comm --help.
| Option | What it does |
|---|---|
-1 | Hide lines only in the first file. |
-2 | Hide lines only in the second file. |
-3 | Hide lines in both files. |
--check-order | Fail on unsorted input even if every line pairs. |
--nocheck-order | Do not check whether the input is sorted. |
--output-delimiter STRING | Separate columns with STRING instead of tabs. |
--total | Print a summary line with the count of each column. |
-z, --zero-terminated | Lines end with NUL instead of newline. |
Related
diffCompare two files or directories line by line and print the changes that turn one into the other.joinJoin lines of two sorted files that share a key field, like a database join on text files.sortSort lines of text alphabetically, numerically, by version or by chosen fields; also merge and de-duplicate.uniqCollapse or report repeated adjacent lines; with sort, count how often each line occurs.- Process substitutionUse <(cmd) and >(cmd) to hand a command's output to a program that wants a file name, keep while-read loops in the current shell, and avoid its timing traps.