Text processing
join
Join lines of two sorted files that share a key field, like a database join on text files.
join [OPTION...] FILE1 FILE2
join matches lines from two files on a common field and prints one combined line per match:
the key, then the remaining fields of the line from the first file, then those from the second.
By default the key is the first field and fields are separated by blanks; -t sets another
separator and -1/-2 choose which field is the key in each file.
The catch that trips everyone: both inputs must be sorted on the key field, in the same
collation join uses (the current locale). join walks the two files in step, like a merge,
which is why it is fast and uses no memory, and why unsorted input gives wrong or missing
results. Sort each input with sort on the same key first, often through process
substitution: join <(sort a) <(sort b).
By default only matching lines are printed (an inner join). -a 1 or -a 2 add unmatched
lines from one side (an outer join) and -v prints only the unmatched ones.
Examples
Join two files on their first field
cache01 cache
db01 database
web01 frontend
web02 frontend
join <(sort data/hosts.txt) roles.txt
cache01 10.0.0.31 cache db01 10.0.0.21 database web01 10.0.0.11 frontend web02 10.0.0.12 frontend
Both inputs are sorted by host name, so every host is matched with its role. Each output line is the key, the IP from the first file and the role from the second.
Unsorted input is an error
cache01 cache
db01 database
web01 frontend
web02 frontend
join data/hosts.txt roles.txt
web01 10.0.0.11 frontend join: data/hosts.txt:3: is not sorted: db01 10.0.0.21 web02 10.0.0.12 frontend join: input is not in sorted order
hosts.txt is not sorted (web01 comes before db01). join still pairs the web servers
but silently skips db01 and cache01, reports the unsorted line on standard error and
exits with status 1. Sort the inputs first.
Join CSV files on a column other than the first
engineering,3rd floor
legal,5th floor
sales,1st floor
support,2nd floor
join -t , -1 4 -2 1 <(tail -n +2 data/users.csv | sort -t , -k 4,4) floors.csv
engineering,1,Alice Moreau,alice@example.com,2021-04-12,3rd floor engineering,3,Chen Wei,chen@example.com,2023-01-20,3rd floor engineering,6,Fatima Haddad,fatima@example.com,2024-09-30,3rd floor sales,2,Bob Okafor,bob@example.com,2019-11-03,1st floor sales,5,Emil Larsen,emil@example.com,2020-02-17,1st floor support,4,Dana Kowalski,dana@example.com,2022-07-08,2nd floor support,7,Gus Ramirez,gus@example.com,2018-05-22,2nd floor
-t , splits on commas, -1 4 uses the fourth field (department) of the users file and
-2 1 the first field of floors.csv. The users are sorted on that same column first. The
key comes first in the output, followed by the other fields.
Choose the output fields
engineering,3rd floor
legal,5th floor
sales,1st floor
support,2nd floor
join -t , -1 4 -2 1 -o 1.2,2.2 <(tail -n +2 data/users.csv | sort -t , -k 4,4) floors.csv
Alice Moreau,3rd floor Chen Wei,3rd floor Fatima Haddad,3rd floor Bob Okafor,1st floor Emil Larsen,1st floor Dana Kowalski,2nd floor Gus Ramirez,2nd floor
-o 1.2,2.2 prints field 2 of the first file (the name) and field 2 of the second (the floor), nothing else.
Keep lines without a match (outer join)
engineering,3rd floor
legal,5th floor
sales,1st floor
support,2nd floor
join -t , -1 4 -2 1 -a 1 -e 'no desk' -o 1.2,2.2 <(tail -n +2 data/users.csv | sort -t , -k 4,4) floors.csv
Alice Moreau,3rd floor Chen Wei,3rd floor Fatima Haddad,3rd floor Hana Sato,no desk Bob Okafor,1st floor Emil Larsen,1st floor Dana Kowalski,2nd floor Gus Ramirez,2nd floor
-a 1 also prints users whose department has no entry in floors.csv: Hana in marketing.
-e 'no desk' fills the missing field, which only works together with -o.
Show only the lines that have no partner
engineering,3rd floor
legal,5th floor
sales,1st floor
support,2nd floor
join -t , -1 4 -2 1 -v 2 <(tail -n +2 data/users.csv | sort -t , -k 4,4) floors.csv
legal,5th floor
-v 2 prints the lines of the second file that matched nothing: there is a floor for legal but no user in that department.
Files with a header line
id,score
1,90
2,85
5,77
join --header -t , data/users.csv scores.csv
id,name,email,dept,joined,score 1,Alice Moreau,alice@example.com,engineering,2021-04-12,90 2,Bob Okafor,bob@example.com,sales,2019-11-03,85 5,Emil Larsen,emil@example.com,sales,2020-02-17,77
--header joins the first lines of both files as headers without checking their order.
The data rows are already sorted by id, so users 1, 2 and 5 get their score appended.
Fill empty fields in an outer join
apple 3
banana 5
cherry 7
apple red
cherry dark-red
date brown
join -a 1 -a 2 -e '-' -o auto a.txt b.txt
apple 3 red banana 5 - cherry 7 dark-red date - brown
Unmatched lines from both files are printed (a full outer join), and -o auto keeps every
row the same shape so -e '-' can fill the gaps: banana has no color, date has no count.
Common pitfalls
Both files must be sorted on the join field with the same collation
joinuses. Sorting withLC_ALL=C sortand joining in another locale (or the reverse) givesnot sortederrors; set the same locale for both commands.Sorting with
sort -k 4(to end of line) instead of-k 4,4can produce an orderjoinrejects when keys are prefixes of each other. Always sort on exactly the key field.joinuses whitespace by default, so a field that contains spaces (like a full name) shifts the field numbers. Use-twith a real separator.Lines with duplicate keys produce every combination (a cross product of the matching lines), just like a SQL join.
GNU vs BSD and macOS
The examples above ran the GNU/Linux version. These are the differences that matter on macOS and the BSDs (written from their documentation, not run here).
--header,-o auto,--check-orderand--nocheck-orderare GNU extensions; BSD/macOSjoinhas-a,-v,-e,-o,-t,-1,-2and-i.
Options worth knowing
A selection, described in our own words. For the full list, see join(1) or join --help.
| Option | What it does |
|---|---|
-t CHAR | Use CHAR as the field separator for input and output. |
-1 FIELD | Join on this field of the first file. |
-2 FIELD | Join on this field of the second file. |
-j FIELD | Join on this field in both files. |
-a FILENUM | Also print unpairable lines from file 1 or 2. |
-v FILENUM | Print only the unpairable lines from file 1 or 2. |
-e STRING | Replace missing fields with STRING (used with -o). |
-o FORMAT | Output fields as a list like 1.2,2.3, or auto. |
-i, --ignore-case | Compare keys case-insensitively. |
--header | Treat the first line of each file as a header and join it without sorting checks. |
--check-order | Fail if the input is not sorted, even when all lines pair. |
--nocheck-order | Do not check whether the input is sorted. |
-z, --zero-terminated | Lines end with NUL instead of newline. |
Related
sortSort lines of text alphabetically, numerically, by version or by chosen fields; also merge and de-duplicate.pasteMerge lines of files side by side, or join all lines of one file into a single line.commCompare two sorted files line by line and show lines unique to each and lines in both.cutPrint selected fields, characters or bytes from each line, such as columns of a CSV or TSV file.awkScan text as records and fields, and run a small program on each line that matches a pattern.- Process substitutionUse <(cmd) and >(cmd) to hand a command's output to a program that wants a file name, keep while-read loops in the current shell, and avoid its timing traps.