Text processing

`awk`

Scan text as records and fields, and run a small program on each line that matches a pattern.

awk [-F SEP] [-v VAR=VALUE] 'PROGRAM' [FILE...]

15 of 15 examples run in Debian 13.7 on 2026-10-31 · gawk 1:5.2.1-2+b1 · manual: awk(1)

awk splits every input line into fields ($1, $2, ... and $0 for the whole line) and runs a program of pattern { action } rules against it. A rule with no pattern runs on every line; a rule with no action prints the line. NR is the current line number, NF the number of fields, and variables spring into existence as 0 or "" the first time you use them.

That makes it the right tool for anything column-shaped: pick fields, filter on a numeric condition, total a column, count occurrences with an associative array, reformat with printf. BEGIN runs before the first line (set separators, print headers) and END after the last (print totals). By default fields are separated by runs of blanks; -F sets another separator such as , or a tab.

On Debian and Ubuntu awk may be gawk (GNU awk, as here) or mawk, the faster default on minimal installs; macOS ships the original BWK awk. Everything below sticks to POSIX awk unless it says otherwise. For one-shot substitutions use sed; for JSON use jq; when a program grows past a dozen lines, a real script language is usually clearer.

Examples

Print one column

$ awk '{print $1}' data/hosts.txt

explain

output

web01
web02
db01
cache01

Each line is split on whitespace and $1 is the first field. Runs of spaces or tabs count as one separator and leading blanks are ignored, which is why awk beats cut on space-aligned text.

Print several fields in a new order

$ awk '{print $2, $1}' data/hosts.txt

explain

output

10.0.0.11 web01
10.0.0.12 web02
10.0.0.21 db01
10.0.0.31 cache01

A comma between items prints them separated by the output field separator OFS, a space by default. Without the comma (print $2 $1) the values are glued together.

Split on a comma with -F

$ awk -F, '{print $2 " <" $3 ">"}' data/users.csv

explain

output

name <email>
Alice Moreau <alice@example.com>
Bob Okafor <bob@example.com>
Chen Wei <chen@example.com>
Dana Kowalski <dana@example.com>
Emil Larsen <emil@example.com>
Fatima Haddad <fatima@example.com>
Gus Ramirez <gus@example.com>
Hana Sato <hana@example.com>

-F, makes the comma the field separator, so $2 is the name and $3 the email. Adjacent strings concatenate. This does not understand quoted CSV fields containing commas; for those use a real CSV parser.

Use NF for the last field

$ awk '{print NF, $NF}' data/poem.txt

explain

output

4 fox
5 dog.
3 sleeps;
4 on.

NF is the number of fields on the line, so $NF is the last field and $(NF-1) the one before it. Punctuation stays attached to words because only blanks separate fields.

Print the lines where a field meets a condition

$ awk -F, '$4 >= 10' data/sales.csv

explain

output

date,region,product,units,price
2026-03-01,north,widget,12,4.50
2026-03-03,west,widget,20,4.50
2026-03-04,north,widget,15,4.50

A pattern with no action prints the matching lines. $4 is the units column, compared as a number on the data lines. Note the header slipped through: units is not a number, so awk compares it as a string against "10", and u sorts after 1. Add NR > 1 && to skip it.

Match a regex against one field

$ awk '$3 == "WARN" || $0 ~ /timeout/' logs/app.log

explain

output

2026-03-14 08:03:45 WARN  failed login for user=alice from 198.51.100.7
2026-03-14 08:03:46 WARN  failed login for user=alice from 198.51.100.7
2026-03-14 08:11:47 WARN  permission denied: DELETE /api/users/3
2026-03-14 08:16:40 ERROR health check failed: database timeout after 5000ms
2026-03-14 08:20:12 WARN  blocked request for /.env from 198.51.100.23

== compares the third field exactly and ~ tests a regex against $0, the whole line. The result is every warning plus the error that mentions a timeout.

Total a column

$ awk -F, 'NR > 1 { units += $4; revenue += $4 * $5 } END { print units, revenue }' data/sales.csv

explain

output

92 743.36

NR > 1 skips the header. Uninitialized variables start at 0, so += just works. The END block runs once after the last line and prints the totals: 92 units and 743.36 in revenue.

Format output with printf

$ awk -F'\t' 'NR > 1 { printf "%-8s %4d  %8.2f\n", $1, $2, $2 * $3 }' data/inventory.tsv

explain

output

widget     12     54.00
gadget      3     59.97
gizmo       5     60.00

-F'\t' splits on tabs. printf takes a C-style format: %-8s is a left-aligned string eight wide, %4d an integer four wide, %8.2f a number with two decimals. Unlike print, printf adds no newline, hence the \n.

Print a header and a footer with BEGIN and END

$ awk 'BEGIN { print "host     ip" } { print $1 "\t" $2; n++ } END { print n " hosts" }' data/hosts.txt

explain

output

host     ip
web01	10.0.0.11
web02	10.0.0.12
db01	10.0.0.21
cache01	10.0.0.31
4 hosts

BEGIN runs before any input is read and END after all of it. n++ counts lines; NR in the END block would give the same number.

Count occurrences with an associative array

$ awk '{count[$1]++} END {for (ip in count) print count[ip], ip}' access.log | sort -rn

explain

output

5 192.0.2.10
4 203.0.113.42
4 198.51.100.7
3 192.0.2.55
2 203.0.113.9
2 198.51.100.23

count[$1]++ uses the client IP as an array key and increments its counter. The for (k in array) loop visits keys in no guaranteed order, so the output is piped to sort -rn to rank the busiest clients first.

Sum a column per group

$ awk -F, 'NR > 1 { total[$2] += $4 * $5 } END { for (r in total) printf "%-6s %7.2f\n", r, total[r] }' data/sales.csv | sort

explain

output

east    207.46
north   181.50
south   220.41
west    133.99

The region ($2) is the key and each line adds units times price to it. This is a GROUP BY region in one line. sort makes the order stable.

Remove duplicate lines without sorting

$ awk '!seen[$0]++' data/fruits.txt

explain

output

banana
apple
cherry
date
elderberry
fig

seen[$0]++ is 0 (false) the first time a line appears and positive afterwards; ! flips that, so only first occurrences print. Unlike sort -u, the original order is kept.

Pass a shell variable in with -v

$ min=8; awk -F, -v min="$min" 'NR > 1 && $4 > min {print $1, $2, $4}' data/sales.csv

explain

output

2026-03-01 north 12
2026-03-03 west 20
2026-03-03 south 9
2026-03-04 north 15

-v name=value sets an awk variable before the program starts. Do this instead of pasting $min into the program text, which breaks on quotes and invites code injection.

Change the output separator

$ awk -F, -v OFS='\t' '{print $1, $2, $4}' data/users.csv | head -4

explain

output

id	name	dept
1	Alice Moreau	engineering
2	Bob Okafor	sales
3	Chen Wei	engineering

OFS is what print puts between comma-separated items. Here CSV becomes tab-separated columns. head -4 keeps the output short.

Print between two patterns or line numbers

$ awk 'NR==3, NR==5 { print NR ": " $3, $4, $5 }' logs/app.log

explain

output

3: WARN failed login
4: WARN failed login
5: INFO user lookup

pattern1, pattern2 is a range: it starts on the first line matching the first pattern and ends on the line matching the second, inclusive. Here the patterns are line numbers 3 and 5, and the action prints the line number with three fields. /start/, /stop/ works the same way with regexes.

Common pitfalls

Wrap the program in single quotes. In double quotes the shell replaces $1 with its own first argument (usually empty) before awk runs.
awk -F, is not a CSV parser: a quoted field such as "Smith, Jr." is split in two. gawk's --csv (5.3+) or a CSV-aware tool is needed for that.
for (k in arr) order is unspecified and differs between awk implementations. Pipe to sort, or in gawk set PROCINFO["sorted_in"].
Assigning to a field ($3 = "x") rebuilds $0 with OFS, so tabs or extra spaces in the original line collapse to single spaces.
Comparisons are numeric only when both sides look like numbers. $2 > 10 on a field like 10kB compares as strings.

GNU vs BSD and macOS

The examples above ran the GNU/Linux version. These are the differences that matter on macOS and the BSDs (written from their documentation, not run here).

awk is gawk on this sandbox, often mawk on minimal Debian/Ubuntu, and the original one-true-awk on macOS and the BSDs. Stick to POSIX features for portable scripts.
gawk-only features include PROCINFO, gensub(), strftime(), asort(), --csv and -i includes. On macOS, install gawk if a script needs them.

Options worth knowing

A selection, described in our own words. For the full list, see awk(1) or awk --help.

Option	What it does
`-F, --field-separator SEP`	Set the input field separator (a character or a regex).
`-v, --assign VAR=VALUE`	Set an awk variable before the program runs.
`-f, --file FILE`	Read the awk program from FILE.
`-e, --source PROGRAM`	Program text, combinable with -f (gawk).
`-i, --include FILE`	Load an awk library file (gawk).
`-P, --posix`	Disable gawk extensions and behave as POSIX awk (gawk).
`-c, --traditional`	Behave like traditional Unix awk (gawk).
`-W OPTION`	Implementation-specific option, such as -W version.
`-V, --version`	Print version information (gawk).

Examples

Common pitfalls

GNU vs BSD and macOS

Options worth knowing

Related