Globbing: bash scripting guide

Globbing (also called pathname expansion) is the shell turning a pattern such as *.log into the list of existing file names that match it. It happens after variable expansion and word splitting, on every unquoted word that contains *, ? or [. The command never sees the pattern, only the resulting names, already sorted.

Pattern	Matches
`*`	any string, including the empty one
`?`	exactly one character
`[abc]`, `[a-h]`	one character from the set or range
`[!abc]` or `[^abc]`	one character not in the set
`[[:digit:]]`, `[[:upper:]]`	one character from a named class

Globs match file names, not text, and two rules set them apart from regular expressions: * never matches a /, and a leading . must be matched explicitly.

The basic patterns

Star, question mark and brackets

$ echo logs/app.log*
> echo logs/app.log.?
> echo data/[a-h]*
> echo photos/[[:upper:]]*

output

logs/app.log logs/app.log.1
logs/app.log.1
data/config.json data/fruits.txt data/hosts.txt
photos/IMG_2041.png

app.log* matches both log files, because * may match nothing. ? needs exactly one character, so only app.log.1 qualifies. The range picks the data/ files whose names start with a to h, and the class [[:upper:]] finds the one photo whose name starts with a capital letter.

Hidden files are skipped by *

$ echo *
> echo .*

output

README.md access.log data file with spaces.txt link-to-notes logs notes.txt photos scripts src todo.txt
.env

* listed everything except .env. A dot at the start of a name has to be matched by a literal dot in the pattern, as .* does. In bash 5.2 .* no longer matches . and .. (the globskipdots option, on by default); older versions included them, which made rm -r .* dangerous.

Quoting turns globbing off

A quoted pattern is just a string. This is why variables holding patterns behave differently depending on quotes: an unquoted $pat is globbed after expansion, a quoted "$pat" is not.

Quoted and unquoted patterns

$ pat='*.md'
> echo $pat
> echo "$pat"
> echo '*.md'

output

README.md
*.md
*.md

Unquoted, $pat expanded to *.md and then globbed to README.md. Quoted, it stayed the literal text. Passing a pattern to find -name therefore needs quotes, so that find receives the pattern instead of the shell's matches.

The no-match pitfall

When a pattern matches nothing, bash by default leaves the word as it is: the literal *.jpg is passed to the command. POSIX requires this, and it surprises everyone at least once, usually in a loop.

A loop over a pattern that matches nothing

$ for f in *.jpg; do
>   echo "processing $f"
> done

output

processing *.jpg

There are no .jpg files, yet the loop ran once, with f set to the literal string *.jpg. Any command inside would fail on a file that does not exist. The classic guard is [ -e "$f" ] || continue as the first line of the loop; the options below are the bash-specific fixes.

nullglob makes a non-matching pattern disappear

$ shopt -s nullglob
> for f in *.jpg; do echo "processing $f"; done
> jpgs=(*.jpg)
> echo "found ${#jpgs[@]} jpg files"

output

found 0 jpg files

With nullglob, *.jpg expands to nothing: the loop body never runs and the array is empty, so counting matches becomes reliable. This is the right setting for loops and arrays.

The flip side of nullglob

$ shopt -s nullglob
> ls *.jpg | head -n 3

output

README.md
access.log
data

The pattern vanished, so ls ran with no arguments and listed the whole current directory, the opposite of what was asked. The same goes for rm -f *.tmp (harmless) or grep foo *.log (which then waits on standard input). Enable nullglob around the loop that needs it, not globally in every script.

failglob turns no match into an error

$ shopt -s failglob
> echo "before"
> ls *.jpg; echo "same line, skipped"
> echo "next line runs, status $?"

output

before
/bin/bash: line 3: no match: *.jpg
next line runs, status 1

With failglob, a pattern with no matches is an error: the command is not run and bash prints no match: *.jpg. In a non-interactive shell it also abandons the rest of that input line, so the second echo never ran; execution continued on the next line with status 1.

dotglob, globstar and nocaseglob

dotglob lets * match names starting with a dot (never . or ..). globstar gives ** its recursive meaning: on its own as a path component, it matches any number of directories, including none. Without globstar, ** behaves exactly like *. nocaseglob makes matching case-insensitive.

Include dotfiles and recurse

$ shopt -s dotglob
> echo *
> echo "--- without globstar:"
> echo **/*.py
> shopt -s globstar
> echo "--- with globstar:"
> echo **/*.py

output

.env README.md access.log data file with spaces.txt link-to-notes logs notes.txt photos scripts src todo.txt
--- without globstar:
src/app.py
--- with globstar:
src/app.py src/lib/helpers.py

With dotglob, .env now appears in *. Without globstar, **/*.py is the same as */*.py and finds only src/app.py, one level down. With it, the search descends into src/lib too. ** also matches hidden directories only when dotglob is on.

extglob: alternation and negation

shopt -s extglob adds five pattern operators that take a |-separated list:

Pattern	Matches
`?(a\\|b)`	zero or one of the patterns
`*(a\\|b)`	zero or more
`+(a\\|b)`	one or more
`@(a\\|b)`	exactly one
`!(a\\|b)`	anything except the patterns

The option changes how bash parses code, so it must be enabled on a line before the one that uses the patterns, and outside the function that uses them. Turning it on and using it on the same line is a syntax error.

Match everything except some extensions

$ shopt -s extglob
> echo data/!(*.json|*.csv)
> echo logs/@(app|error).log
> echo logs/app.log?(.1)
> echo photos/+([a-z]).png

output

data/fruits.txt data/hosts.txt data/inventory.tsv data/numbers.txt data/poem.txt data/settings.conf data/windows.txt
logs/app.log logs/error.log
logs/app.log logs/app.log.1
photos/beach.png photos/forest.png

!(*.json|*.csv) lists the data files that are neither JSON nor CSV, something plain globs cannot express. @(app|error) picks one of two names, ?(.1) makes the suffix optional, and +([a-z]) requires one or more lowercase letters, which rules out IMG_2041.png. The same operators work inside [[ $x == pattern ]] (where bash enables them automatically) and in ${var#pattern}.

Sort order and file names that look like options

Matches are sorted according to the locale's collation. In this sandbox (C.UTF-8) that is byte order, so uppercase names come before lowercase ones; in en_US.UTF-8 they would be interleaved. Do not rely on the order for anything important.

A glob can also expand to a name starting with -, which the command then reads as an option. Prefix patterns with ./ or end the options with --.

A file named -l changes what ls does

Setup run before this example

mkdir uploads && touch uploads/-l uploads/a.txt uploads/b.txt

$ cd uploads
> ls *
> echo "--- with ./"
> ls ./*

output

-rw-r--r-- 1 dev dev 0 Oct 31 20:22 a.txt
-rw-r--r-- 1 dev dev 0 Oct 31 20:22 b.txt
--- with ./
./-l
./a.txt
./b.txt

* expanded to -l a.txt b.txt, and ls took -l as its long-listing option, so the file named -l was never listed. With ./* every match starts with ./, nothing looks like an option, and all three files appear. ls -- * would work too.

The basic patterns

Quoting turns globbing off

The no-match pitfall

dotglob, globstar and nocaseglob

extglob: alternation and negation

Sort order and file names that look like options

Related