How to use sed in shell scripts | Scripting & Automation Tutorial

How to Use sed in Shell Scripts The `sed` (Stream Editor) command is one of the most powerful text processing tools available in Unix-like systems. This comprehensive guide will teach you how to effectively use `sed` in shell scripts for automating text manipulation tasks, from basic substitutions to complex pattern matching and file processing operations. Table of Contents 1. [Introduction to sed](#introduction-to-sed) 2. [Prerequisites](#prerequisites) 3. [Understanding sed Basics](#understanding-sed-basics) 4. [sed Syntax and Command Structure](#sed-syntax-and-command-structure) 5. [Basic sed Operations](#basic-sed-operations) 6. [Advanced sed Techniques](#advanced-sed-techniques) 7. [Using sed in Shell Scripts](#using-sed-in-shell-scripts) 8. [Real-World Examples](#real-world-examples) 9. [Common Issues and Troubleshooting](#common-issues-and-troubleshooting) 10. [Best Practices and Tips](#best-practices-and-tips) 11. [Conclusion](#conclusion) Introduction to sed The `sed` command is a stream editor that performs basic text transformations on an input stream (a file or input from a pipeline). Unlike interactive text editors, `sed` is designed to work non-interactively, making it perfect for shell scripts and automated text processing tasks. It reads text line by line, applies specified operations, and outputs the results. Key advantages of using `sed` in shell scripts include: - Non-interactive processing: Perfect for automation - Memory efficient: Processes large files without loading them entirely into memory - Powerful pattern matching: Uses regular expressions for complex text operations - Versatile operations: Supports substitution, deletion, insertion, and more - Cross-platform compatibility: Available on virtually all Unix-like systems Prerequisites Before diving into `sed` usage in shell scripts, ensure you have: - Basic understanding of Unix/Linux command line - Familiarity with shell scripting fundamentals - Knowledge of regular expressions (recommended but not required) - Access to a Unix-like system (Linux, macOS, or WSL on Windows) - A text editor for creating shell scripts Understanding sed Basics How sed Works The `sed` command follows a simple workflow: 1. Read: Reads one line from input stream into pattern space 2. Execute: Applies specified commands to the pattern space 3. Output: Prints the pattern space (unless suppressed) 4. Repeat: Continues with the next line until end of input Pattern Space and Hold Space - Pattern Space: The internal buffer where `sed` holds the current line being processed - Hold Space: An auxiliary buffer for temporary storage during complex operations sed Syntax and Command Structure The basic syntax of `sed` is: ```bash sed [OPTIONS] 'COMMAND' [INPUT-FILE...] ``` Common Options | Option | Description | |--------|-------------| | `-n` | Suppress automatic printing of pattern space | | `-e` | Add script commands | | `-f` | Read script from file | | `-i` | Edit files in-place | | `-r` or `-E` | Use extended regular expressions | Command Structure sed commands follow this pattern: ``` [ADDRESS]COMMAND[OPTIONS] ``` Where: - ADDRESS: Specifies which lines to process - COMMAND: The operation to perform - OPTIONS: Additional parameters for the command Basic sed Operations 1. Substitution (s Command) The substitution command is the most commonly used `sed` operation: ```bash Basic substitution syntax sed 's/pattern/replacement/flags' file.txt Example: Replace first occurrence of 'old' with 'new' on each line sed 's/old/new/' file.txt Replace all occurrences (global flag) sed 's/old/new/g' file.txt Case-insensitive replacement sed 's/old/new/gi' file.txt ``` 2. Deletion (d Command) Remove specific lines from output: ```bash Delete line 3 sed '3d' file.txt Delete lines 2 to 5 sed '2,5d' file.txt Delete lines matching pattern sed '/pattern/d' file.txt Delete empty lines sed '/^$/d' file.txt ``` 3. Print (p Command) Print specific lines (usually used with -n option): ```bash Print line 5 sed -n '5p' file.txt Print lines 10 to 20 sed -n '10,20p' file.txt Print lines matching pattern sed -n '/pattern/p' file.txt ``` 4. Insertion and Appending Add text before (i) or after (a) specific lines: ```bash Insert text before line 3 sed '3i\This is inserted text' file.txt Append text after line matching pattern sed '/pattern/a\This is appended text' file.txt ``` Advanced sed Techniques 1. Address Ranges Use various addressing methods to target specific lines: ```bash Line numbers sed '1,5s/old/new/g' file.txt # Lines 1-5 sed '10,$s/old/new/g' file.txt # Line 10 to end Pattern ranges sed '/start/,/end/s/old/new/g' file.txt # Between patterns Step addressing sed '1~2s/old/new/g' file.txt # Every other line starting from 1 ``` 2. Multiple Commands Execute multiple `sed` commands: ```bash Using -e option sed -e 's/old/new/g' -e 's/foo/bar/g' file.txt Using semicolon sed 's/old/new/g; s/foo/bar/g' file.txt Using newlines sed ' s/old/new/g s/foo/bar/g ' file.txt ``` 3. Backreferences Use captured groups in replacements: ```bash Capture and reuse patterns sed 's/$[0-9]$-$[0-9]$/\2-\1/g' file.txt Swap words sed 's/$.$ $.$/\2 \1/' file.txt ``` 4. Hold Space Operations Advanced pattern manipulation using hold space: ```bash Copy pattern space to hold space sed 'h' file.txt Append pattern space to hold space sed 'H' file.txt Copy hold space to pattern space sed 'g' file.txt Exchange pattern and hold spaces sed 'x' file.txt ``` Using sed in Shell Scripts 1. Basic Script Integration Here's how to incorporate `sed` into shell scripts: ```bash #!/bin/bash Simple sed usage in script input_file="data.txt" output_file="processed_data.txt" Process file and save to new file sed 's/old_value/new_value/g' "$input_file" > "$output_file" echo "Processing complete. Output saved to $output_file" ``` 2. Using Variables in sed When using shell variables in `sed` commands, use double quotes: ```bash #!/bin/bash search_term="error" replacement="warning" file_path="/var/log/application.log" Use variables in sed command sed "s/$search_term/$replacement/g" "$file_path" For complex patterns, consider using different delimiters sed "s|$search_term|$replacement|g" "$file_path" ``` 3. In-Place Editing Modify files directly using the `-i` option: ```bash #!/bin/bash config_file="/etc/myapp/config.conf" Backup and modify in-place sed -i.backup 's/debug=false/debug=true/g' "$config_file" Modify without backup (use with caution) sed -i 's/old_setting/new_setting/g' "$config_file" ``` 4. Processing Multiple Files Handle multiple files in a loop: ```bash #!/bin/bash Process all .txt files in directory for file in *.txt; do if [[ -f "$file" ]]; then echo "Processing $file..." sed -i 's/old/new/g' "$file" fi done echo "All files processed." ``` Real-World Examples 1. Log File Processing Script ```bash #!/bin/bash Script to clean and process log files log_file="/var/log/application.log" clean_log="/tmp/cleaned.log" Remove empty lines, timestamps, and sensitive data sed -e '/^$/d' \ -e 's/[0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\} [0-9]\{2\}:[0-9]\{2\}:[0-9]\{2\}//g' \ -e 's/password=[^[:space:]]*/password=/g' \ "$log_file" > "$clean_log" echo "Log cleaned and saved to $clean_log" ``` 2. Configuration File Update Script ```bash #!/bin/bash Update database configuration config_file="database.conf" new_host="new-db-server.com" new_port="5432" Update configuration values sed -i \ -e "s/^host=.*/host=$new_host/" \ -e "s/^port=.*/port=$new_port/" \ -e 's/^#$ssl_enabled$/\1/' \ "$config_file" echo "Configuration updated successfully" ``` 3. CSV Data Processing ```bash #!/bin/bash Process CSV file: normalize data and fix formatting input_csv="raw_data.csv" output_csv="processed_data.csv" sed -e 's/,[ \t]*/, /g' \ -e 's/[ \t]*,/,/g' \ -e 's/""/NULL/g' \ -e 's/\r$//' \ "$input_csv" > "$output_csv" echo "CSV processing complete" ``` 4. HTML Processing Script ```bash #!/bin/bash Remove HTML tags and clean up text html_file="input.html" text_file="output.txt" sed -e 's/<[^>]*>//g' \ -e 's/ / /g' \ -e 's/&/\&/g' \ -e 's/<//g' \ -e '/^$/d' \ "$html_file" > "$text_file" echo "HTML converted to plain text" ``` 5. Advanced Text Transformation ```bash #!/bin/bash Convert text format and restructure data data_file="employee_data.txt" Transform "Last, First (ID)" to "ID: First Last" sed -E 's/^([^,]+), ([^(]+) $([^)]+)$$/\3: \2 \1/' "$data_file" | sed 's/ */ /g' | # Remove extra spaces sed 's/ $//' # Remove trailing spaces echo "Data transformation complete" ``` Common Issues and Troubleshooting 1. Escaping Special Characters Problem: Special characters in patterns cause unexpected behavior. Solution: Properly escape metacharacters: ```bash Escape dots, asterisks, brackets, etc. sed 's/\./DOT/g' file.txt # Literal dot sed 's/\*/ASTERISK/g' file.txt # Literal asterisk sed 's/\[/LEFT_BRACKET/g' file.txt # Literal bracket ``` 2. Variable Expansion Issues Problem: Variables not expanding correctly in `sed` commands. Solution: Use proper quoting: ```bash Wrong - single quotes prevent variable expansion sed 's/$old_value/$new_value/g' file.txt Correct - use double quotes sed "s/$old_value/$new_value/g" file.txt Alternative - use different delimiter to avoid conflicts sed "s|$old_value|$new_value|g" file.txt ``` 3. In-Place Editing Errors Problem: File corruption or permission errors with `-i` option. Solution: Always create backups and check permissions: ```bash Create backup before in-place editing sed -i.backup 's/old/new/g' file.txt Check if file is writable if [[ -w "$file" ]]; then sed -i 's/old/new/g' "$file" else echo "Error: Cannot write to $file" exit 1 fi ``` 4. Regular Expression Compatibility Problem: Different `sed` versions support different regex features. Solution: Use portable patterns or specify extended regex: ```bash Portable basic regex sed 's/[0-9]\{3\}/XXX/g' file.txt Extended regex (GNU sed) sed -E 's/[0-9]{3}/XXX/g' file.txt ``` 5. Handling Large Files Problem: Performance issues with very large files. Solution: Use appropriate addressing and consider alternatives: ```bash Process only necessary lines sed -n '1000,2000p' large_file.txt For very large files, consider split processing split -l 10000 large_file.txt chunk_ for chunk in chunk_*; do sed 's/old/new/g' "$chunk" > "processed_$chunk" done ``` Best Practices and Tips 1. Script Organization - Use meaningful variable names for patterns and files - Comment complex `sed` commands - Break long command chains into multiple steps for readability ```bash #!/bin/bash Well-organized sed usage readonly INPUT_FILE="$1" readonly OUTPUT_FILE="$2" readonly SEARCH_PATTERN="old_value" readonly REPLACEMENT="new_value" Validate input if [[ ! -f "$INPUT_FILE" ]]; then echo "Error: Input file not found" >&2 exit 1 fi Process file with clear, commented steps sed -e "s/$SEARCH_PATTERN/$REPLACEMENT/g" \ # Replace values -e '/^#/d' \ # Remove comments -e '/^$/d' \ # Remove empty lines "$INPUT_FILE" > "$OUTPUT_FILE" ``` 2. Error Handling Always implement proper error handling: ```bash #!/bin/bash process_file() { local input_file="$1" local output_file="$2" # Check if sed command succeeds if sed 's/old/new/g' "$input_file" > "$output_file"; then echo "File processed successfully" return 0 else echo "Error processing file" >&2 return 1 fi } Usage with error checking if ! process_file "input.txt" "output.txt"; then exit 1 fi ``` 3. Testing and Validation - Test `sed` commands on sample data before applying to production files - Use `-n` and `p` commands to preview changes - Validate output format and content ```bash #!/bin/bash Preview changes before applying echo "Preview of changes:" sed -n 's/old/new/gp' input.txt | head -10 read -p "Apply changes? (y/N): " confirm if [[ "$confirm" == "y" || "$confirm" == "Y" ]]; then sed -i.backup 's/old/new/g' input.txt echo "Changes applied. Backup saved as input.txt.backup" fi ``` 4. Performance Optimization - Use specific addressing to limit processing scope - Combine multiple operations when possible - Consider using `awk` or other tools for complex operations ```bash Efficient: combine operations sed -e 's/old/new/g' -e '/pattern/d' -e 's/foo/bar/g' file.txt Less efficient: separate commands sed 's/old/new/g' file.txt | sed '/pattern/d' | sed 's/foo/bar/g' ``` 5. Portability Considerations Write scripts that work across different systems: ```bash #!/bin/bash Detect sed version and adjust accordingly if sed --version >/dev/null 2>&1; then # GNU sed SED_EXTENDED="sed -E" SED_INPLACE="sed -i" else # BSD sed (macOS) SED_EXTENDED="sed -E" SED_INPLACE="sed -i ''" fi Use variables for compatibility $SED_EXTENDED 's/[0-9]{3}/XXX/g' file.txt ``` Conclusion The `sed` command is an indispensable tool for shell script automation, offering powerful text processing capabilities that can handle everything from simple substitutions to complex data transformations. By mastering the techniques covered in this guide, you can: - Automate repetitive text processing tasks - Create robust scripts for log file analysis - Process configuration files dynamically - Handle large-scale data transformations efficiently Next Steps To further enhance your `sed` skills: 1. Practice with real data: Apply these techniques to actual files in your work environment 2. Explore advanced features: Learn about `sed`'s programming constructs like branching and loops 3. Combine with other tools: Integrate `sed` with `awk`, `grep`, and other Unix utilities 4. Study regular expressions: Deepen your pattern matching knowledge 5. Read system documentation: Consult your system's `sed` manual (`man sed`) for specific features Remember that while `sed` is powerful, it's not always the best tool for every text processing task. Consider alternatives like `awk` for complex field-based operations or `perl` for very complex pattern matching requirements. The key is choosing the right tool for each specific task while building a comprehensive toolkit for text processing automation. With consistent practice and application of these best practices, you'll be able to leverage `sed` effectively in your shell scripts, creating more efficient and maintainable automation solutions.