opscape

Bash scripting guide · part 29 of 32

String manipulation

Length, substrings, prefix and suffix removal, search and replace, case, trimming, splitting and joining, done with bash's own expansions instead of external tools.

Bash can do most everyday string work by itself, through parameter expansion. That matters for two reasons. Each $(echo "$x" | sed ...) starts at least one extra process, which is noticeable inside a loop that runs thousands of times. And the built-in forms do not suffer from echo quirks with strings like -n or backslashes.

The opposite also holds: for transforming a whole file, sed, awk or tr over the file is far faster than a bash while read loop that edits each line. Use the expansions on variables, the tools on streams.

Length and substrings

Measure a string and cut pieces out of it

s="Hello, World"
echo "length: ${#s}"
echo "from offset 7: ${s:7}"
echo "first 5: ${s:0:5}"
echo "last 5: ${s: -5}"
echo "drop 3 at each end: ${s:3:-3}"
output
length: 12
from offset 7: World
first 5: Hello
last 5: World
drop 3 at each end: lo, Wo

${s:offset:length} counts from zero. A negative offset counts from the end, but it needs a space (or parentheses) after the colon: ${s:-5} would be the "default value" expansion instead. A negative length means "stop this many characters before the end".

Length counts characters, according to the locale

word="café"
echo "UTF-8 locale: ${#word} characters, $(printf %s "$word" | wc -c) bytes"
LC_ALL=C
echo "C locale: ${#word}"
output
UTF-8 locale: 4 characters, 5 bytes
C locale: 5

In the sandbox's C.UTF-8 locale, é is one character but two bytes. Under LC_ALL=C bash counts bytes, so the same string has length 5. Substring offsets follow the same rule, so set the locale deliberately when slicing non-ASCII text.

Removing a prefix or suffix

Four operators strip a glob pattern from either end. One symbol removes the shortest match, the doubled symbol the longest. # works at the front, % at the back (on a US keyboard # is left of %, the same order as start and end).

Take a path apart without basename and dirname

f=/var/log/demo/app.log.1
echo "file name:   ${f##*/}"
echo "directory:   ${f%/*}"
echo "last ext:    ${f##*.}"
echo "drop ext:    ${f%.*}"
echo "drop all ext: ${f%%.*}"
output
file name:   app.log.1
directory:   /var/log/demo
last ext:    1
drop ext:    /var/log/demo/app.log
drop all ext: /var/log/demo/app

##*/ removes the longest prefix ending in a slash, leaving the file name. %/* removes the shortest suffix starting with one, leaving the directory. %.* drops only .1, while %%.* drops everything from the first dot. Unlike dirname, ${f%/*} returns the input unchanged when there is no slash, so check for that if it matters.

Search and replace

${var/pattern/replacement} replaces the first match, ${var//pattern/replacement} every match. /# anchors the pattern to the start, /% to the end. The pattern is a glob, not a regular expression.

Replace the first, every, leading or trailing match

p="2026-03-14 08:00:01"
echo "${p/-//}"
echo "${p//-//}"
echo "${p// /T}"
echo "${p/#2026/26}"
echo "${p/%:01/:00}"
echo "${p//[0-9]/#}"
output
2026/03-14 08:00:01
2026/03/14 08:00:01
2026-03-14T08:00:01
26-03-14 08:00:01
2026-03-14 08:00:00
####-##-## ##:##:##

Note the slashes: ${p/-//} reads as pattern -, replacement /. The last line uses a bracket expression, so every digit becomes #. An empty replacement deletes: ${p//-} removes all dashes.

Case conversion

Upper case, lower case, first letter

s="hello, World"
echo "${s^^}"
echo "${s,,}"
echo "${s^}"
echo "${s~~}"
declare -u loud="set once, upper forever"
echo "$loud"
output
HELLO, WORLD
hello, world
Hello, World
HELLO, wORLD
SET ONCE, UPPER FOREVER

^^ and ,, convert the whole string, a single ^ or , only the first character, and ~~ swaps the case of each letter. declare -u (or -l) makes a variable convert everything assigned to it. These need bash 4 or later; the bash 3.2 that macOS ships does not have them, so use tr '[:lower:]' '[:upper:]' there.

Trimming whitespace

There is no trim operator, but two nested expansions do it: find the run of leading whitespace, then strip exactly that.

Trim leading and trailing whitespace

x=$'  \t padded value  '
y=${x#"${x%%[![:space:]]*}"}
y=${y%"${y##*[![:space:]]}"}
echo "[$y]"
read -r z <<< "$x"
echo "[$z]"
output
[padded value]
[padded value]

${x%%[![:space:]]*} is everything before the first non-space character, the leading whitespace; removing that prefix trims the left side, and the mirror image trims the right. The inner expansion is quoted so its characters are matched literally. The shortcut on the last two lines works because read strips leading and trailing IFS whitespace, but it also processes only the first line of the string.

Testing what a string contains

[[ ... == pattern ]] matches against a glob when the right side is unquoted, and =~ matches an extended regular expression and stores the captures in BASH_REMATCH.

Starts with, contains, and regex captures

v="v2.4.1-rc1"
[[ $v == v* ]] && echo "starts with v"
[[ $v == *-rc* ]] && echo "is a release candidate"
[[ $v == "*-rc*" ]] || echo "quoted pattern is compared literally"
if [[ $v =~ ^v([0-9]+)\.([0-9]+)\.([0-9]+) ]]; then
  echo "major=${BASH_REMATCH[1]} minor=${BASH_REMATCH[2]} patch=${BASH_REMATCH[3]}"
fi
output
starts with v
is a release candidate
quoted pattern is compared literally
major=2 minor=4 patch=1

Quoting the right side of == turns the pattern into a literal string, which is how you compare against text that contains *. For =~, leave the regex unquoted, or store it in a variable and write [[ $v =~ $re ]]; quoting it makes it a literal match too.

Splitting and joining

Split a string on a delimiter into an array

line="alice,engineering,,2023-01-09"
IFS=, read -r -a fields <<< "$line"
echo "${#fields[@]} fields"
printf '[%s]\n' "${fields[@]}"
output
4 fields
[alice]
[engineering]
[]
[2023-01-09]

Setting IFS=, only for the read command splits on commas without changing IFS for the rest of the script. The empty field between the two commas is kept, because comma is not a whitespace character. -a stores the pieces in an array. This does not understand CSV quoting; for real CSV use a proper parser.

Join array elements with a separator

hosts=(web01 web02 db01)
(IFS=,; echo "${hosts[*]}")
printf -v joined '%s, ' "${hosts[@]}"
echo "${joined%, }"
output
web01,web02,db01
web01, web02, db01

"${arr[*]}" joins the elements with the first character of IFS; the parentheses keep the changed IFS inside a subshell. For a multi-character separator, let printf repeat its format for every element and remove the trailing separator afterwards.

Formatting with printf -v

printf -v var writes the formatted result into a variable instead of standard output, without a subshell. Widths and padding work as in C: %-10s left-aligns in 10 columns, %05d pads with zeros.

Pad, align and zero-fill

for row in "web01 4" "cache01 16" "db01 8"; do
  read -r name cpus <<< "$row"
  printf -v line '%-8s|%4d|%03d' "$name" "$cpus" "$cpus"
  echo "$line"
done
output
web01   |   4|004
cache01 |  16|016
db01    |   8|008

Each line has the name left-aligned in 8 columns, the number right-aligned in 4, and the same number zero-padded to 3 digits. Note that printf widths count bytes, so non-ASCII text can misalign columns.

Quick reference

Expansion Result
${#s} length
${s:o:l} substring from offset o, length l
${s#p} ${s##p} remove shortest / longest prefix matching p
${s%p} ${s%%p} remove shortest / longest suffix matching p
${s/p/r} ${s//p/r} replace first / all matches
${s^^} ${s,,} upper / lower case
${s@Q} quoted for reuse as shell input

The default-value forms (${v:-x}, ${v:?msg} and friends) are on Parameter expansion.

Related