How to use loops in shell scripts
How to Use Loops in Shell Scripts
Shell scripting is a powerful tool for automating repetitive tasks in Unix and Linux environments. Among the most essential constructs in shell scripting are loops, which allow you to execute a set of commands multiple times based on specific conditions. Whether you're processing files, iterating through data, or automating system administration tasks, understanding how to effectively use loops will significantly enhance your shell scripting capabilities.
This comprehensive guide will walk you through the three main types of loops available in shell scripts: `for` loops, `while` loops, and `until` loops. You'll learn their syntax, practical applications, and best practices to help you write efficient and maintainable shell scripts.
Table of Contents
1. [Prerequisites](#prerequisites)
2. [Understanding Loop Fundamentals](#understanding-loop-fundamentals)
3. [For Loops](#for-loops)
4. [While Loops](#while-loops)
5. [Until Loops](#until-loops)
6. [Loop Control Statements](#loop-control-statements)
7. [Nested Loops](#nested-loops)
8. [Practical Examples and Use Cases](#practical-examples-and-use-cases)
9. [Common Issues and Troubleshooting](#common-issues-and-troubleshooting)
10. [Best Practices and Tips](#best-practices-and-tips)
11. [Conclusion](#conclusion)
Prerequisites
Before diving into shell script loops, ensure you have:
- Basic understanding of shell scripting concepts
- Familiarity with command-line interface
- Access to a Unix/Linux system or terminal
- Knowledge of basic shell commands and variables
- Understanding of file permissions and script execution
- Basic text editor skills (vim, nano, or similar)
Understanding Loop Fundamentals
Loops are control structures that repeat a block of code until a specific condition is met. In shell scripting, loops help automate repetitive tasks and process collections of data efficiently. The three primary types of loops in shell scripts are:
1. For loops: Execute a set of commands for each item in a list
2. While loops: Continue executing commands while a condition remains true
3. Until loops: Execute commands until a condition becomes true
Each loop type serves different purposes and is suited for specific scenarios. Understanding when and how to use each type will make your scripts more efficient and readable.
For Loops
The `for` loop is the most commonly used loop in shell scripting. It iterates over a list of items, executing the loop body for each item. The basic syntax is:
```bash
for variable in list
do
commands
done
```
Basic For Loop Examples
Example 1: Simple List Iteration
```bash
#!/bin/bash
Iterate through a simple list of words
for fruit in apple banana cherry date
do
echo "I like $fruit"
done
```
Output:
```
I like apple
I like banana
I like cherry
I like date
```
Example 2: Iterating Through Numbers
```bash
#!/bin/bash
Iterate through a range of numbers
for number in {1..5}
do
echo "Number: $number"
done
```
Output:
```
Number: 1
Number: 2
Number: 3
Number: 4
Number: 5
```
Example 3: C-style For Loop
```bash
#!/bin/bash
C-style for loop syntax
for ((i=1; i<=5; i++))
do
echo "Iteration: $i"
done
```
Advanced For Loop Techniques
Iterating Through Files
```bash
#!/bin/bash
Process all .txt files in current directory
for file in *.txt
do
if [ -f "$file" ]; then
echo "Processing file: $file"
# Add your file processing commands here
wc -l "$file"
fi
done
```
Using Command Substitution
```bash
#!/bin/bash
Iterate through command output
for user in $(cat /etc/passwd | cut -d: -f1)
do
echo "User: $user"
done
```
Array Iteration
```bash
#!/bin/bash
Define an array and iterate through it
servers=("web01" "web02" "db01" "cache01")
for server in "${servers[@]}"
do
echo "Checking server: $server"
ping -c 1 "$server" > /dev/null 2>&1
if [ $? -eq 0 ]; then
echo "$server is online"
else
echo "$server is offline"
fi
done
```
While Loops
The `while` loop continues executing as long as the specified condition remains true. It's particularly useful when you don't know in advance how many iterations will be needed.
Basic While Loop Syntax
```bash
while [ condition ]
do
commands
done
```
While Loop Examples
Example 1: Counter-based While Loop
```bash
#!/bin/bash
Simple counter loop
counter=1
while [ $counter -le 5 ]
do
echo "Count: $counter"
counter=$((counter + 1))
done
```
Example 2: Reading File Line by Line
```bash
#!/bin/bash
Read a file line by line
filename="data.txt"
while IFS= read -r line
do
echo "Processing: $line"
done < "$filename"
```
Example 3: Menu System
```bash
#!/bin/bash
Simple menu system using while loop
choice=""
while [ "$choice" != "quit" ]
do
echo "Menu Options:"
echo "1. List files"
echo "2. Show date"
echo "3. Show users"
echo "Type 'quit' to exit"
read -p "Enter your choice: " choice
case $choice in
1) ls -la ;;
2) date ;;
3) who ;;
quit) echo "Goodbye!" ;;
*) echo "Invalid option" ;;
esac
done
```
Example 4: Monitoring System Resources
```bash
#!/bin/bash
Monitor disk usage until it exceeds threshold
threshold=80
while true
do
usage=$(df / | awk 'NR==2 {print $5}' | sed 's/%//')
if [ $usage -gt $threshold ]; then
echo "Warning: Disk usage is ${usage}%"
break
fi
echo "Disk usage: ${usage}% - OK"
sleep 60 # Check every minute
done
```
Until Loops
The `until` loop is the opposite of the `while` loop. It continues executing until the specified condition becomes true. It's less commonly used but can make certain scripts more readable.
Basic Until Loop Syntax
```bash
until [ condition ]
do
commands
done
```
Until Loop Examples
Example 1: Wait for File Creation
```bash
#!/bin/bash
Wait until a specific file is created
filename="important_file.txt"
until [ -f "$filename" ]
do
echo "Waiting for $filename to be created..."
sleep 2
done
echo "File $filename has been created!"
```
Example 2: Service Availability Check
```bash
#!/bin/bash
Wait until a service is available
service_port=80
host="localhost"
until nc -z "$host" "$service_port" 2>/dev/null
do
echo "Waiting for service on $host:$service_port..."
sleep 5
done
echo "Service is now available!"
```
Example 3: User Input Validation
```bash
#!/bin/bash
Keep asking for valid input until received
valid_input=""
until [ -n "$valid_input" ]
do
read -p "Enter a non-empty value: " input
if [ -n "$input" ]; then
valid_input="$input"
else
echo "Input cannot be empty. Please try again."
fi
done
echo "You entered: $valid_input"
```
Loop Control Statements
Shell scripts provide two important loop control statements that give you fine-grained control over loop execution:
Break Statement
The `break` statement immediately exits the current loop:
```bash
#!/bin/bash
Using break to exit loop early
for i in {1..10}
do
if [ $i -eq 6 ]; then
echo "Breaking at $i"
break
fi
echo "Number: $i"
done
echo "Loop ended"
```
Continue Statement
The `continue` statement skips the rest of the current iteration and moves to the next:
```bash
#!/bin/bash
Using continue to skip iterations
for i in {1..10}
do
if [ $((i % 2)) -eq 0 ]; then
continue # Skip even numbers
fi
echo "Odd number: $i"
done
```
Practical Example: File Processing with Controls
```bash
#!/bin/bash
Process files with error handling
for file in *.log
do
if [ ! -f "$file" ]; then
echo "No log files found"
break
fi
if [ ! -r "$file" ]; then
echo "Cannot read $file, skipping..."
continue
fi
echo "Processing $file..."
# Process the file here
grep "ERROR" "$file" > "${file}.errors"
done
```
Nested Loops
Nested loops allow you to place one loop inside another, enabling complex data processing scenarios:
Example 1: Matrix Processing
```bash
#!/bin/bash
Create a multiplication table
echo "Multiplication Table:"
for i in {1..5}
do
for j in {1..5}
do
result=$((i * j))
printf "%3d " $result
done
echo # New line after each row
done
```
Example 2: Directory and File Processing
```bash
#!/bin/bash
Process files in multiple directories
directories=("logs" "data" "config")
file_extensions=("txt" "log" "conf")
for dir in "${directories[@]}"
do
if [ -d "$dir" ]; then
echo "Processing directory: $dir"
for ext in "${file_extensions[@]}"
do
for file in "$dir"/*."$ext"
do
if [ -f "$file" ]; then
echo " Found file: $file"
# Process file here
fi
done
done
fi
done
```
Practical Examples and Use Cases
System Administration Tasks
Example 1: Backup Script
```bash
#!/bin/bash
Backup important directories
backup_dirs=("/etc" "/home" "/var/www")
backup_destination="/backup/$(date +%Y%m%d)"
mkdir -p "$backup_destination"
for dir in "${backup_dirs[@]}"
do
if [ -d "$dir" ]; then
echo "Backing up $dir..."
tar -czf "$backup_destination/$(basename $dir).tar.gz" "$dir"
if [ $? -eq 0 ]; then
echo "Successfully backed up $dir"
else
echo "Failed to backup $dir"
fi
else
echo "Directory $dir does not exist, skipping..."
fi
done
```
Example 2: Log Rotation Script
```bash
#!/bin/bash
Rotate log files older than 7 days
log_directory="/var/log/myapp"
max_age=7
find "$log_directory" -name "*.log" -mtime +$max_age | while read -r logfile
do
echo "Rotating old log file: $logfile"
gzip "$logfile"
mv "${logfile}.gz" "${log_directory}/archive/"
done
```
Data Processing Tasks
Example 3: CSV File Processing
```bash
#!/bin/bash
Process CSV file and extract specific data
csv_file="employees.csv"
output_file="email_list.txt"
Skip header line and process each record
tail -n +2 "$csv_file" | while IFS=',' read -r name email department salary
do
if [ "$department" = "IT" ]; then
echo "$email" >> "$output_file"
echo "Added IT employee: $name ($email)"
fi
done
```
Network Administration
Example 4: Network Connectivity Check
```bash
#!/bin/bash
Check connectivity to multiple servers
servers=("google.com" "github.com" "stackoverflow.com")
timeout=3
for server in "${servers[@]}"
do
echo "Testing connection to $server..."
if ping -c 1 -W $timeout "$server" > /dev/null 2>&1; then
echo "✓ $server is reachable"
else
echo "✗ $server is unreachable"
fi
done
```
Common Issues and Troubleshooting
Issue 1: Infinite Loops
Problem: Loop never terminates, causing script to hang.
Solution: Always ensure your loop condition will eventually become false.
```bash
Bad example - infinite loop
counter=1
while [ $counter -le 10 ]
do
echo $counter
# Forgot to increment counter!
done
Good example - proper termination
counter=1
while [ $counter -le 10 ]
do
echo $counter
counter=$((counter + 1)) # Always update loop variable
done
```
Issue 2: File Name Issues with Spaces
Problem: File names containing spaces cause loops to break incorrectly.
Solution: Use proper quoting and IFS handling.
```bash
Bad example
for file in $(ls *.txt)
do
echo $file # Will break on spaces
done
Good example
for file in *.txt
do
if [ -f "$file" ]; then
echo "$file" # Properly quoted
fi
done
```
Issue 3: Variable Scope in Subshells
Problem: Variables modified inside loops (especially with pipes) don't persist.
```bash
Bad example - counter won't be updated
counter=0
cat file.txt | while read line
do
counter=$((counter + 1))
done
echo $counter # Will still be 0
```
Solution: Use different approaches to avoid subshells.
```bash
Good example - avoid subshell
counter=0
while read line
do
counter=$((counter + 1))
done < file.txt
echo $counter # Will show correct count
```
Issue 4: Command Substitution Performance
Problem: Using command substitution in loop conditions can be slow.
```bash
Inefficient - command runs every iteration
while [ $(date +%H) -lt 17 ]
do
# Commands here
sleep 60
done
```
Solution: Store command results in variables when possible.
```bash
More efficient
current_hour=$(date +%H)
while [ $current_hour -lt 17 ]
do
# Commands here
sleep 60
current_hour=$(date +%H) # Update only when needed
done
```
Best Practices and Tips
1. Use Meaningful Variable Names
```bash
Poor naming
for i in *.txt
do
echo $i
done
Better naming
for text_file in *.txt
do
echo "Processing: $text_file"
done
```
2. Always Quote Variables
```bash
Risky - can break with special characters
for file in $file_list
do
process $file
done
Safe - properly quoted
for file in "$file_list"
do
process "$file"
done
```
3. Check for Empty Lists
```bash
Check if files exist before processing
txt_files=(*.txt)
if [ -f "${txt_files[0]}" ]; then
for file in "${txt_files[@]}"
do
echo "Processing: $file"
done
else
echo "No .txt files found"
fi
```
4. Use Arrays for Complex Data
```bash
Define arrays for better organization
declare -a servers=("web01" "web02" "db01")
declare -a ports=(80 443 3306)
for i in "${!servers[@]}"
do
server="${servers[$i]}"
port="${ports[$i]}"
echo "Checking $server on port $port"
done
```
5. Implement Error Handling
```bash
Add error checking in loops
for config_file in /etc/myapp/*.conf
do
if [ ! -r "$config_file" ]; then
echo "Warning: Cannot read $config_file" >&2
continue
fi
if ! validate_config "$config_file"; then
echo "Error: Invalid configuration in $config_file" >&2
exit 1
fi
echo "Configuration $config_file is valid"
done
```
6. Use Functions for Complex Loop Bodies
```bash
Define function for complex operations
process_log_file() {
local log_file="$1"
local error_count=$(grep -c "ERROR" "$log_file")
local warning_count=$(grep -c "WARNING" "$log_file")
echo "File: $log_file"
echo " Errors: $error_count"
echo " Warnings: $warning_count"
}
Use function in loop
for log_file in /var/log/app/*.log
do
if [ -f "$log_file" ]; then
process_log_file "$log_file"
fi
done
```
7. Consider Performance for Large Datasets
```bash
For large file processing, use efficient tools
Instead of reading line by line in shell
while IFS= read -r line
do
# Process line
done < large_file.txt
Consider using awk, sed, or other tools
awk '{print "Processing: " $0}' large_file.txt
```
8. Document Complex Loops
```bash
#!/bin/bash
Multi-level backup script with retention policy
Processes multiple backup types with different retention periods
declare -A backup_config=(
["daily"]="7"
["weekly"]="4"
["monthly"]="12"
)
for backup_type in "${!backup_config[@]}"
do
retention_days="${backup_config[$backup_type]}"
echo "Processing $backup_type backups (retain ${retention_days} days)"
# Find and remove old backups
find "/backup/$backup_type" -name "*.tar.gz" -mtime +$retention_days | while read -r old_backup
do
echo "Removing old backup: $old_backup"
rm "$old_backup"
done
done
```
Conclusion
Mastering loops in shell scripts is essential for creating efficient, maintainable automation solutions. Throughout this guide, we've explored the three main types of loops—for, while, and until—along with their practical applications, common pitfalls, and best practices.
Key takeaways from this comprehensive guide include:
- For loops are ideal for iterating over known lists or ranges
- While loops excel when you need to continue based on dynamic conditions
- Until loops provide readable alternatives for negative conditions
- Proper variable quoting and error handling prevent common issues
- Loop control statements (break and continue) provide fine-grained execution control
- Nested loops enable complex data processing scenarios
As you continue developing your shell scripting skills, remember to:
- Always test your loops with edge cases
- Implement proper error handling and logging
- Consider performance implications for large datasets
- Document complex loop logic for future maintenance
- Use meaningful variable names and consistent coding style
With these fundamentals and best practices in mind, you're well-equipped to leverage the power of loops in your shell scripts, whether you're automating system administration tasks, processing data files, or building complex workflow automation solutions.
Continue practicing with real-world scenarios, and don't hesitate to combine loops with other shell scripting constructs like functions, conditional statements, and external tools to create robust, professional-grade automation scripts.