How to create simple shell scripts
How to Create Simple Shell Scripts
Shell scripting is one of the most powerful and essential skills for anyone working with Unix-like operating systems, including Linux and macOS. Whether you're a system administrator, developer, or power user, learning to create shell scripts can dramatically improve your productivity by automating repetitive tasks, managing system operations, and streamlining complex workflows.
This comprehensive guide will take you from complete beginner to confident shell script creator, covering everything from basic syntax to advanced techniques, practical examples, and professional best practices.
Table of Contents
1. [What Are Shell Scripts?](#what-are-shell-scripts)
2. [Prerequisites and Requirements](#prerequisites-and-requirements)
3. [Getting Started with Your First Script](#getting-started-with-your-first-script)
4. [Essential Shell Scripting Components](#essential-shell-scripting-components)
5. [Variables and Data Types](#variables-and-data-types)
6. [Control Structures](#control-structures)
7. [Functions in Shell Scripts](#functions-in-shell-scripts)
8. [Practical Examples and Use Cases](#practical-examples-and-use-cases)
9. [Input and Output Operations](#input-and-output-operations)
10. [Error Handling and Debugging](#error-handling-and-debugging)
11. [Best Practices and Professional Tips](#best-practices-and-professional-tips)
12. [Common Issues and Troubleshooting](#common-issues-and-troubleshooting)
13. [Advanced Techniques](#advanced-techniques)
14. [Conclusion and Next Steps](#conclusion-and-next-steps)
What Are Shell Scripts?
Shell scripts are text files containing a series of commands that can be executed by a shell interpreter. Think of them as automated command sequences that perform tasks you would otherwise execute manually in the terminal. Shell scripts can range from simple one-liners that save you a few keystrokes to complex programs that manage entire system operations.
The most common shell for scripting is Bash (Bourne Again Shell), which is the default shell on most Linux distributions and macOS systems. Other popular shells include Zsh, Fish, and the original Bourne shell (sh).
Benefits of Shell Scripting
- Automation: Eliminate repetitive manual tasks
- Consistency: Ensure tasks are performed the same way every time
- Efficiency: Save time and reduce human error
- Integration: Combine multiple tools and commands seamlessly
- Scheduling: Run scripts automatically using cron jobs
- Portability: Scripts can run on any compatible Unix-like system
Prerequisites and Requirements
Before diving into shell scripting, ensure you have the following:
System Requirements
- A Unix-like operating system (Linux, macOS, or Windows with WSL)
- Access to a terminal or command-line interface
- A text editor (nano, vim, VS Code, or any preferred editor)
- Basic familiarity with command-line operations
Essential Knowledge
- Understanding of basic terminal commands (`ls`, `cd`, `cp`, `mv`, `rm`)
- File system navigation concepts
- Basic understanding of file permissions
- Familiarity with text editing in your chosen editor
Checking Your Shell
First, identify which shell you're using:
```bash
echo $SHELL
```
This command will display your current shell. Most systems use `/bin/bash` or `/bin/zsh`.
Getting Started with Your First Script
Let's create your first shell script to understand the fundamental structure and execution process.
Step 1: Create the Script File
Open your terminal and create a new file:
```bash
nano hello_world.sh
```
Step 2: Add the Shebang Line
Every shell script should start with a shebang (`#!`) line that specifies which interpreter to use:
```bash
#!/bin/bash
```
Step 3: Add Your Commands
Add the following content to your script:
```bash
#!/bin/bash
This is a comment - my first shell script
echo "Hello, World!"
echo "Today's date is: $(date)"
echo "Current user: $USER"
```
Step 4: Save and Exit
In nano, press `Ctrl+X`, then `Y`, then `Enter` to save and exit.
Step 5: Make the Script Executable
Before running the script, you need to make it executable:
```bash
chmod +x hello_world.sh
```
Step 6: Execute the Script
Run your script using one of these methods:
```bash
Method 1: Direct execution
./hello_world.sh
Method 2: Explicit shell invocation
bash hello_world.sh
```
You should see output similar to:
```
Hello, World!
Today's date is: Mon Oct 23 14:30:25 PDT 2023
Current user: john
```
Essential Shell Scripting Components
Understanding the core components of shell scripts is crucial for writing effective automation tools.
Comments
Comments make your code readable and maintainable:
```bash
#!/bin/bash
Single-line comment
echo "This line will execute"
: '
Multi-line comment
Everything between the quotes
will be ignored
'
echo "This will also execute"
```
Exit Status
Every command returns an exit status (0 for success, non-zero for failure):
```bash
#!/bin/bash
Check if a file exists
if [ -f "myfile.txt" ]; then
echo "File exists"
exit 0
else
echo "File does not exist"
exit 1
fi
```
Command Substitution
Capture command output for use in your script:
```bash
#!/bin/bash
Using $() syntax (recommended)
current_date=$(date)
echo "Current date: $current_date"
Using backticks (older syntax)
user_count=`who | wc -l`
echo "Users logged in: $user_count"
```
Variables and Data Types
Variables are essential for storing and manipulating data in shell scripts.
Declaring and Using Variables
```bash
#!/bin/bash
Variable assignment (no spaces around =)
name="John Doe"
age=30
pi=3.14159
Using variables
echo "Name: $name"
echo "Age: ${age} years old"
echo "Pi value: $pi"
Read-only variables
readonly CONFIG_FILE="/etc/myapp.conf"
```
Environment Variables
Access system and environment variables:
```bash
#!/bin/bash
echo "Home directory: $HOME"
echo "Current path: $PATH"
echo "Shell: $SHELL"
echo "Username: $USER"
Setting environment variables
export MY_VAR="Hello World"
echo "My variable: $MY_VAR"
```
Arrays
Bash supports both indexed and associative arrays:
```bash
#!/bin/bash
Indexed arrays
fruits=("apple" "banana" "orange" "grape")
echo "First fruit: ${fruits[0]}"
echo "All fruits: ${fruits[@]}"
echo "Number of fruits: ${#fruits[@]}"
Adding elements
fruits+=("mango")
Associative arrays (Bash 4+)
declare -A colors
colors[red]="#FF0000"
colors[green]="#00FF00"
colors[blue]="#0000FF"
echo "Red color code: ${colors[red]}"
```
Special Variables
Shell scripts have access to special built-in variables:
```bash
#!/bin/bash
echo "Script name: $0"
echo "First argument: $1"
echo "Second argument: $2"
echo "All arguments: $@"
echo "Number of arguments: $#"
echo "Process ID: $$"
echo "Exit status of last command: $?"
```
Control Structures
Control structures allow you to create logic and flow in your scripts.
Conditional Statements
If-Then-Else
```bash
#!/bin/bash
read -p "Enter a number: " number
if [ $number -gt 10 ]; then
echo "Number is greater than 10"
elif [ $number -eq 10 ]; then
echo "Number is exactly 10"
else
echo "Number is less than 10"
fi
```
Case Statements
```bash
#!/bin/bash
read -p "Enter your choice (1-3): " choice
case $choice in
1)
echo "You selected option 1"
;;
2)
echo "You selected option 2"
;;
3)
echo "You selected option 3"
;;
*)
echo "Invalid choice"
;;
esac
```
Loops
For Loops
```bash
#!/bin/bash
Loop through a list
for fruit in apple banana orange; do
echo "Processing: $fruit"
done
Loop through files
for file in *.txt; do
echo "Found text file: $file"
done
C-style for loop
for ((i=1; i<=5; i++)); do
echo "Count: $i"
done
```
While Loops
```bash
#!/bin/bash
counter=1
while [ $counter -le 5 ]; do
echo "Iteration: $counter"
((counter++))
done
Reading file line by line
while IFS= read -r line; do
echo "Line: $line"
done < "input.txt"
```
Until Loops
```bash
#!/bin/bash
counter=1
until [ $counter -gt 5 ]; do
echo "Counter: $counter"
((counter++))
done
```
Functions in Shell Scripts
Functions help organize code and promote reusability.
Defining and Calling Functions
```bash
#!/bin/bash
Function definition
greet_user() {
echo "Hello, $1!"
echo "Welcome to our system."
}
Function with return value
calculate_square() {
local number=$1
local result=$((number * number))
echo $result
}
Function calls
greet_user "Alice"
square_result=$(calculate_square 5)
echo "Square of 5 is: $square_result"
```
Advanced Function Features
```bash
#!/bin/bash
Function with multiple parameters
create_backup() {
local source_dir=$1
local backup_dir=$2
local timestamp=$(date +"%Y%m%d_%H%M%S")
if [ ! -d "$source_dir" ]; then
echo "Error: Source directory does not exist"
return 1
fi
mkdir -p "$backup_dir"
cp -r "$source_dir" "${backup_dir}/backup_${timestamp}"
echo "Backup created successfully"
return 0
}
Usage
create_backup "/home/user/documents" "/backup"
```
Practical Examples and Use Cases
Let's explore real-world scenarios where shell scripts prove invaluable.
Example 1: System Monitoring Script
```bash
#!/bin/bash
System monitoring script
echo "=== System Monitoring Report ==="
echo "Generated on: $(date)"
echo
CPU usage
echo "CPU Usage:"
top -bn1 | grep "Cpu(s)" | awk '{print $2 $3}' | sed 's/%us,/% user,/'
Memory usage
echo
echo "Memory Usage:"
free -h | grep "Mem:" | awk '{print "Used: " $3 " / Total: " $2}'
Disk usage
echo
echo "Disk Usage:"
df -h | grep -E '^/dev/' | awk '{print $1 ": " $3 "/" $2 " (" $5 " used)"}'
Network connections
echo
echo "Active Network Connections:"
netstat -tuln | grep LISTEN | wc -l | awk '{print $1 " listening ports"}'
```
Example 2: File Organization Script
```bash
#!/bin/bash
File organization script
organize_downloads() {
local download_dir="$HOME/Downloads"
# Create directories if they don't exist
mkdir -p "$download_dir"/{Images,Documents,Videos,Archives,Others}
# Move files based on extension
cd "$download_dir" || exit 1
# Images
for ext in jpg jpeg png gif bmp; do
mv *.$ext Images/ 2>/dev/null
done
# Documents
for ext in pdf doc docx txt; do
mv *.$ext Documents/ 2>/dev/null
done
# Videos
for ext in mp4 avi mkv mov; do
mv *.$ext Videos/ 2>/dev/null
done
# Archives
for ext in zip tar gz rar; do
mv *.$ext Archives/ 2>/dev/null
done
echo "Files organized successfully!"
}
organize_downloads
```
Example 3: Automated Backup Script
```bash
#!/bin/bash
Automated backup script with rotation
BACKUP_SOURCE="/home/user/important_data"
BACKUP_DEST="/backup"
MAX_BACKUPS=7
DATE=$(date +"%Y%m%d_%H%M%S")
Function to create backup
create_backup() {
echo "Starting backup at $(date)"
# Create backup directory
backup_name="backup_$DATE"
backup_path="$BACKUP_DEST/$backup_name"
# Create backup
if tar -czf "$backup_path.tar.gz" -C "$(dirname $BACKUP_SOURCE)" "$(basename $BACKUP_SOURCE)"; then
echo "Backup created: $backup_path.tar.gz"
else
echo "Error: Backup failed"
exit 1
fi
}
Function to rotate old backups
rotate_backups() {
cd "$BACKUP_DEST" || exit 1
# Count existing backups
backup_count=$(ls -1 backup_*.tar.gz 2>/dev/null | wc -l)
if [ $backup_count -gt $MAX_BACKUPS ]; then
# Remove oldest backups
ls -1t backup_*.tar.gz | tail -n +$((MAX_BACKUPS + 1)) | xargs rm -f
echo "Rotated old backups"
fi
}
Main execution
mkdir -p "$BACKUP_DEST"
create_backup
rotate_backups
echo "Backup process completed at $(date)"
```
Input and Output Operations
Effective input/output handling is crucial for interactive and automated scripts.
Reading User Input
```bash
#!/bin/bash
Simple input
read -p "Enter your name: " username
echo "Hello, $username!"
Silent input (for passwords)
read -s -p "Enter password: " password
echo
echo "Password entered (hidden)"
Input with timeout
if read -t 10 -p "Enter something (10 seconds): " input; then
echo "You entered: $input"
else
echo "Timeout reached"
fi
Multiple inputs
read -p "Enter first name: " first_name
read -p "Enter last name: " last_name
echo "Full name: $first_name $last_name"
```
File Operations
```bash
#!/bin/bash
Reading files
while IFS= read -r line; do
echo "Processing: $line"
done < "input.txt"
Writing to files
echo "Log entry: $(date)" >> logfile.txt
Checking file properties
filename="test.txt"
if [ -f "$filename" ]; then
echo "File exists"
echo "Size: $(stat -c%s "$filename") bytes"
echo "Modified: $(stat -c%y "$filename")"
fi
```
Command Line Arguments
```bash
#!/bin/bash
Process command line arguments
if [ $# -eq 0 ]; then
echo "Usage: $0 [file2] [file3] ..."
exit 1
fi
echo "Processing $# files:"
for file in "$@"; do
if [ -f "$file" ]; then
echo "Processing: $file ($(wc -l < "$file") lines)"
else
echo "Warning: $file not found"
fi
done
```
Error Handling and Debugging
Robust error handling and debugging techniques are essential for reliable scripts.
Error Handling Strategies
```bash
#!/bin/bash
Enable strict error handling
set -euo pipefail
Custom error handling function
handle_error() {
echo "Error occurred in script $0 at line $1"
echo "Exit code: $2"
exit $2
}
Trap errors
trap 'handle_error $LINENO $?' ERR
Function with error checking
safe_copy() {
local source=$1
local dest=$2
if [ ! -f "$source" ]; then
echo "Error: Source file '$source' does not exist"
return 1
fi
if ! cp "$source" "$dest"; then
echo "Error: Failed to copy '$source' to '$dest'"
return 1
fi
echo "Successfully copied '$source' to '$dest'"
return 0
}
Usage with error handling
if safe_copy "important.txt" "/backup/"; then
echo "Backup completed successfully"
else
echo "Backup failed"
exit 1
fi
```
Debugging Techniques
```bash
#!/bin/bash
Enable debug mode
set -x # Print commands as they execute
Debug function
debug() {
if [ "${DEBUG:-0}" = "1" ]; then
echo "DEBUG: $*" >&2
fi
}
Usage
debug "Starting processing"
Conditional debugging
DEBUG=1 ./myscript.sh
Logging function
log() {
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a script.log
}
log "Script started"
log "Processing files..."
log "Script completed"
```
Best Practices and Professional Tips
Following best practices ensures your scripts are maintainable, reliable, and professional.
Code Organization and Style
```bash
#!/bin/bash
Script: system_maintenance.sh
Purpose: Perform routine system maintenance tasks
Author: Your Name
Date: 2023-10-23
Version: 1.0
Enable strict mode
set -euo pipefail
Global variables
readonly SCRIPT_NAME="${0##*/}"
readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
readonly LOG_FILE="/var/log/maintenance.log"
Configuration
BACKUP_DIR="/backup"
RETENTION_DAYS=30
DEBUG=${DEBUG:-0}
Function definitions
usage() {
cat << EOF
Usage: $SCRIPT_NAME [OPTIONS]
OPTIONS:
-h, --help Show this help message
-v, --verbose Enable verbose output
-d, --debug Enable debug mode
-b, --backup-dir Specify backup directory (default: $BACKUP_DIR)
EXAMPLES:
$SCRIPT_NAME --verbose
$SCRIPT_NAME --backup-dir /custom/backup
EOF
}
Logging function
log() {
local level=$1
shift
echo "[$(date '+%Y-%m-%d %H:%M:%S')] [$level] $*" | tee -a "$LOG_FILE"
}
Main function
main() {
log "INFO" "Starting system maintenance"
# Your main logic here
log "INFO" "System maintenance completed"
}
Parse command line arguments
while [[ $# -gt 0 ]]; do
case $1 in
-h|--help)
usage
exit 0
;;
-v|--verbose)
set -x
shift
;;
-d|--debug)
DEBUG=1
shift
;;
-b|--backup-dir)
BACKUP_DIR="$2"
shift 2
;;
*)
echo "Unknown option: $1"
usage
exit 1
;;
esac
done
Execute main function
main "$@"
```
Security Considerations
```bash
#!/bin/bash
Secure shell scripting practices
1. Validate input
validate_input() {
local input=$1
# Check for dangerous characters
if [[ "$input" =~ [;\&\|] ]]; then
echo "Error: Invalid characters in input"
return 1
fi
# Check input length
if [ ${#input} -gt 100 ]; then
echo "Error: Input too long"
return 1
fi
return 0
}
2. Use full paths for commands
readonly RM_CMD="/bin/rm"
readonly CP_CMD="/bin/cp"
readonly FIND_CMD="/usr/bin/find"
3. Quote variables properly
safe_remove() {
local file="$1"
if [ -f "$file" ]; then
"$RM_CMD" "$file"
fi
}
4. Create temporary files securely
temp_file=$(mktemp) || exit 1
trap "rm -f '$temp_file'" EXIT
5. Check permissions before operations
check_permissions() {
local dir=$1
if [ ! -w "$dir" ]; then
echo "Error: No write permission for directory: $dir"
return 1
fi
}
```
Common Issues and Troubleshooting
Understanding common pitfalls helps you write more robust scripts and debug issues quickly.
Permission Issues
```bash
#!/bin/bash
Common permission problems and solutions
Problem: Script not executable
Solution: Check and fix permissions
check_executable() {
local script=$1
if [ ! -x "$script" ]; then
echo "Making script executable: $script"
chmod +x "$script"
fi
}
Problem: Cannot write to directory
Solution: Check write permissions
safe_write() {
local file=$1
local content=$2
local dir=$(dirname "$file")
if [ ! -w "$dir" ]; then
echo "Error: Cannot write to directory: $dir"
return 1
fi
echo "$content" > "$file"
}
```
Variable and Quoting Issues
```bash
#!/bin/bash
Common variable problems
Problem: Spaces in filenames
Wrong way:
for file in $(ls *.txt); do
echo $file # This will break with spaces
done
Correct way:
for file in *.txt; do
echo "$file" # Always quote variables
done
Problem: Uninitialized variables
Solution: Use parameter expansion with defaults
process_file() {
local filename=${1:-"default.txt"} # Use default if not provided
local output_dir=${2:-"/tmp"} # Use default directory
echo "Processing: $filename in $output_dir"
}
```
Path and Environment Issues
```bash
#!/bin/bash
Handle path and environment issues
Problem: Command not found
Solution: Check if command exists
check_command() {
local cmd=$1
if ! command -v "$cmd" >/dev/null 2>&1; then
echo "Error: Command '$cmd' not found"
echo "Please install $cmd or check your PATH"
return 1
fi
}
Problem: Different behavior on different systems
Solution: Detect system and adapt
detect_system() {
case "$(uname -s)" in
Linux*) SYSTEM=Linux;;
Darwin*) SYSTEM=Mac;;
CYGWIN*) SYSTEM=Cygwin;;
MINGW*) SYSTEM=MinGw;;
*) SYSTEM="UNKNOWN:$(uname -s)"
esac
echo "Detected system: $SYSTEM"
}
```
Debugging Common Script Failures
```bash
#!/bin/bash
Debugging utilities
Enable comprehensive debugging
debug_mode() {
set -x # Print commands
set -e # Exit on error
set -u # Error on undefined variables
set -o pipefail # Pipe failures
}
Check script syntax without execution
syntax_check() {
local script=$1
if bash -n "$script"; then
echo "Syntax OK: $script"
else
echo "Syntax errors found in: $script"
return 1
fi
}
Trace function calls
trace_calls() {
PS4='+ ${FUNCNAME[0]:+${FUNCNAME[0]}():}line ${LINENO}: '
set -x
}
```
Advanced Techniques
Once you've mastered the basics, these advanced techniques will help you create more sophisticated scripts.
Process Management
```bash
#!/bin/bash
Advanced process management
Run commands in parallel
parallel_processing() {
local jobs=()
# Start background jobs
for i in {1..5}; do
{
echo "Processing job $i"
sleep $((RANDOM % 5 + 1))
echo "Job $i completed"
} &
jobs+=($!)
done
# Wait for all jobs to complete
for job in "${jobs[@]}"; do
wait "$job"
done
echo "All parallel jobs completed"
}
Process monitoring
monitor_process() {
local process_name=$1
local max_wait=60
local count=0
while [ $count -lt $max_wait ]; do
if pgrep "$process_name" > /dev/null; then
echo "Process $process_name is running"
return 0
fi
sleep 1
((count++))
done
echo "Process $process_name not found after ${max_wait}s"
return 1
}
```
Signal Handling
```bash
#!/bin/bash
Advanced signal handling
Cleanup function
cleanup() {
echo "Cleaning up..."
# Remove temporary files
rm -f /tmp/script_temp_*
# Kill background processes
jobs -p | xargs -r kill
echo "Cleanup completed"
exit 0
}
Signal handlers
handle_sigint() {
echo "Received SIGINT (Ctrl+C)"
cleanup
}
handle_sigterm() {
echo "Received SIGTERM"
cleanup
}
Set up signal traps
trap handle_sigint SIGINT
trap handle_sigterm SIGTERM
trap cleanup EXIT
Long-running process simulation
echo "Starting long-running process (Press Ctrl+C to interrupt)"
for i in {1..100}; do
echo "Processing step $i"
sleep 1
done
```
Configuration Management
```bash
#!/bin/bash
Configuration file handling
Default configuration
declare -A CONFIG
CONFIG[backup_dir]="/backup"
CONFIG[retention_days]="30"
CONFIG[log_level]="INFO"
CONFIG[email_notify]="false"
Load configuration from file
load_config() {
local config_file=${1:-"$HOME/.myscript.conf"}
if [ -f "$config_file" ]; then
echo "Loading configuration from: $config_file"
while IFS='=' read -r key value; do
# Skip comments and empty lines
[[ $key =~ ^[[:space:]]*# ]] && continue
[[ -z $key ]] && continue
# Remove quotes from value
value=$(echo "$value" | sed 's/^["'\'']\|["'\'']$//g')
CONFIG[$key]=$value
done < "$config_file"
else
echo "Configuration file not found, using defaults"
fi
}
Save configuration
save_config() {
local config_file=${1:-"$HOME/.myscript.conf"}
echo "# Configuration file for myscript" > "$config_file"
echo "# Generated on $(date)" >> "$config_file"
echo >> "$config_file"
for key in "${!CONFIG[@]}"; do
echo "${key}=${CONFIG[$key]}" >> "$config_file"
done
echo "Configuration saved to: $config_file"
}
Usage
load_config
echo "Backup directory: ${CONFIG[backup_dir]}"
echo "Retention days: ${CONFIG[retention_days]}"
```
Conclusion and Next Steps
Congratulations! You've now learned the fundamentals of shell scripting, from creating your first simple script to implementing advanced techniques for process management, error handling, and configuration management. Shell scripting is a powerful skill that will serve you well in system administration, development, and automation tasks.
Key Takeaways
1. Start Simple: Begin with basic scripts and gradually add complexity
2. Follow Best Practices: Use proper error handling, quoting, and documentation
3. Test Thoroughly: Always test your scripts in safe environments first
4. Security Matters: Validate inputs and use secure coding practices
5. Maintainability: Write clean, well-documented code that others can understand
Next Steps for Continued Learning
1. Practice Regularly: Create scripts for your daily tasks to build proficiency
2. Study Advanced Topics:
- Regular expressions and text processing
- Network programming with shell scripts
- Integration with APIs using curl and jq
- Database interactions from shell scripts
3. Explore Related Technologies:
- Learn about cron jobs for script scheduling
- Study systemd for service management
- Explore configuration management tools like Ansible
- Investigate containerization with Docker
4. Join Communities: Participate in forums, contribute to open-source projects, and learn from experienced developers
5. Build Real Projects: Create practical tools like system monitors, backup solutions, or deployment scripts
Resources for Further Learning
- Advanced Bash-Scripting Guide
- Shell scripting communities and forums
- Open-source projects using shell scripts
- System administration courses and certifications
Remember that mastering shell scripting is an iterative process. Start with simple automation tasks in your daily work, and gradually tackle more complex challenges. Each script you write will improve your skills and understanding of system automation.
Shell scripting remains one of the most practical and immediately useful programming skills you can develop. Whether you're managing servers, automating deployments, or simply organizing your personal files, the techniques covered in this guide will help you work more efficiently and effectively.
Happy scripting!