Bash Parameter Expansion: String Manipulation Without External Commands

yazar

Originally published at harshit.cloud on 2024-12-17.

TIL: Bash Parameter Expansion: String Manipulation Without External Commands

Learned that Bash has built-in string manipulation that’s way faster than calling sed, awk, or cut.

the old way (slow)

filename="document.pdf"
name=$(echo "$filename" | sed 's/.[^.]*$//')  # document
ext=$(echo "$filename" | sed 's/^.*.//')      # pdf

Each echo | sed spawns a new process. Slow in loops.

the new way (fast)

filename="document.pdf"
name="${filename%.*}"    # document
ext="${filename##*.}"    # pdf

No external processes. Pure Bash.

common patterns

remove file extension

file="archive.tar.gz"
echo "${file%.*}"       # archive.tar
echo "${file%%.*}"      # archive (remove all extensions)

% = remove from end (shortest match)
%% = remove from end (longest match)

get file extension

file="archive.tar.gz"
echo "${file#*.}"       # tar.gz
echo "${file##*.}"      # gz (last extension only)

# = remove from start (shortest match)
## = remove from start (longest match)

string replacement

path="/home/user/documents/file.txt"

# Replace first occurrence
echo "${path/user/admin}"        # /home/admin/documents/file.txt

# Replace all occurrences
echo "${path//o/0}"              # /h0me/user/d0cuments/file.txt

# Replace at start
echo "${path/#/home//root}"    # /root/user/documents/file.txt

# Replace at end
echo "${path/%.txt/.md}"         # /home/user/documents/file.md

default values

# Use default if variable is unset or empty
echo "${VAR:-default}"

# Assign default if unset
echo "${VAR:=default}"

# Error if unset
echo "${VAR:?Variable is required}"

# Use alternate value if set
echo "${VAR:+value_if_set}"

substring extraction

text="Hello World"
echo "${text:0:5}"       # Hello (from pos 0, length 5)
echo "${text:6}"         # World (from pos 6 to end)
echo "${text: -5}"       # World (last 5 chars, note the space!)
echo "${text::-6}"       # Hello (remove last 6 chars)

string length

text="Hello World"
echo "${#text}"          # 11

case conversion

text="Hello World"
echo "${text^^}"         # HELLO WORLD (all uppercase)
echo "${text,,}"         # hello world (all lowercase)
echo "${text^}"          # Hello World (first char uppercase)
echo "${text,}"          # hello World (first char lowercase)

real-world example

Before (slow with multiple processes): 

#!/bin/bash
for file in *.log; do
    name=$(basename "$file" .log)
    date=$(echo "$name" | cut -d'-' -f1)
    gzip "$file"
    mv "${file}.gz" "archive-${date}.gz"
done

After (fast, pure Bash): 

#!/bin/bash
for file in *.log; do
    name="${file%.log}"      # Remove .log
    date="${name%%-*}"       # Get everything before first -
    gzip "$file"
    mv "${file}.gz" "archive-${date}.gz"
done

cheat sheet

Pattern Effect Example
${var%pattern} Remove shortest match from end ${file%.txt}
${var%%pattern} Remove longest match from end ${file%%.*}
${var#pattern} Remove shortest match from start ${file#*/}
${var##pattern} Remove longest match from start ${file##*/}
${var/old/new} Replace first occurrence ${text/foo/bar}
${var//old/new} Replace all occurrences ${text//foo/bar}
${var:offset:length} Substring ${text:0:5}
${#var} Length ${#text}
${var^^} Uppercase ${text^^}
${var,,} Lowercase ${text,,}

why this matters

In a script processing 10,000 files:

  • With external commands: ~5 minutes
  • With parameter expansion: ~10 seconds

That’s a 30x speedup.

one gotcha

Be careful with spaces in substring extraction: 

text="Hello"
echo "${text: -3}"    # llo (CORRECT - note the space)
echo "${text:-3}"     # Hello (WRONG - default value syntax!)

This simple trick has made my shell scripts so much faster. No more unnecessary sed/awk/cut calls.