How to edit text streams → sed - Text Processing Guide

How to Edit Text Streams → sed Table of Contents 1. [Introduction](#introduction) 2. [Prerequisites](#prerequisites) 3. [Understanding sed Fundamentals](#understanding-sed-fundamentals) 4. [Basic sed Syntax and Commands](#basic-sed-syntax-and-commands) 5. [Text Substitution and Replacement](#text-substitution-and-replacement) 6. [Pattern Matching and Addressing](#pattern-matching-and-addressing) 7. [Advanced sed Operations](#advanced-sed-operations) 8. [Practical Examples and Use Cases](#practical-examples-and-use-cases) 9. [Working with Multiple Files](#working-with-multiple-files) 10. [Troubleshooting Common Issues](#troubleshooting-common-issues) 11. [Best Practices and Professional Tips](#best-practices-and-professional-tips) 12. [Conclusion](#conclusion) Introduction The `sed` (Stream Editor) command is one of the most powerful and versatile text processing tools available in Unix-like operating systems. This non-interactive stream editor allows you to perform complex text transformations, substitutions, deletions, and insertions on text streams and files without opening them in a traditional text editor. Whether you're a system administrator managing configuration files, a developer processing log files, or a data analyst cleaning datasets, mastering `sed` will significantly enhance your text processing capabilities. This comprehensive guide will take you from basic concepts to advanced techniques, providing practical examples and real-world applications. By the end of this article, you'll understand how to: - Execute basic and advanced sed commands - Perform complex text substitutions and transformations - Use regular expressions effectively with sed - Handle multiple files and batch operations - Implement sed in shell scripts and automation workflows - Troubleshoot common issues and optimize performance Prerequisites Before diving into sed operations, ensure you have: System Requirements - Unix-like operating system (Linux, macOS, BSD) or Windows with WSL - Access to a terminal or command-line interface - Basic familiarity with command-line operations Knowledge Prerequisites - Understanding of basic file operations and text manipulation - Familiarity with regular expressions (helpful but not mandatory) - Basic knowledge of shell scripting concepts Tools and Setup ```bash Verify sed installation sed --version Create test files for practice echo -e "Hello World\nThis is line 2\nAnother line\nFinal line" > test.txt echo -e "apple,banana,cherry\ndog,elephant,fox\n123,456,789" > data.csv ``` Understanding sed Fundamentals What is sed? `sed` is a stream editor that processes text line by line, applying specified commands to each line as it reads through the input. Unlike interactive editors, sed operates automatically according to predetermined instructions, making it ideal for batch processing and automation. Key Characteristics Non-interactive Processing: sed processes text automatically without user intervention during execution. Stream-based Operation: Text is processed as a continuous stream, line by line, without loading entire files into memory. Pattern Space Concept: sed maintains a pattern space (buffer) where it temporarily stores and manipulates each line. Command-based Interface: Operations are specified through single-letter commands combined with addresses and options. sed Workflow ``` Input Stream → Pattern Space → Apply Commands → Output Stream ``` 1. Read a line from input into pattern space 2. Apply all applicable commands to the line 3. Output the modified line (unless suppressed) 4. Repeat for the next line Basic sed Syntax and Commands Command Structure The basic sed command structure follows this pattern: ```bash sed [options] 'command' [file...] sed [options] -e 'command1' -e 'command2' [file...] sed [options] -f script_file [file...] ``` Essential Options | Option | Description | Example | |--------|-------------|---------| | `-e` | Execute multiple commands | `sed -e 's/old/new/' -e 's/foo/bar/'` | | `-n` | Suppress automatic output | `sed -n '2p' file.txt` | | `-i` | Edit files in-place | `sed -i 's/old/new/' file.txt` | | `-f` | Read commands from file | `sed -f script.sed file.txt` | | `-r` or `-E` | Use extended regex | `sed -r 's/[0-9]+/NUM/' file.txt` | Core sed Commands Print Command (p) ```bash Print specific lines sed -n '2p' test.txt # Print line 2 sed -n '1,3p' test.txt # Print lines 1-3 sed -n '/pattern/p' test.txt # Print lines matching pattern ``` Delete Command (d) ```bash Delete specific lines sed '2d' test.txt # Delete line 2 sed '1,3d' test.txt # Delete lines 1-3 sed '/pattern/d' test.txt # Delete lines matching pattern ``` Substitute Command (s) ```bash Basic substitution sed 's/old/new/' test.txt # Replace first occurrence per line sed 's/old/new/g' test.txt # Replace all occurrences sed 's/old/new/2' test.txt # Replace second occurrence per line ``` Text Substitution and Replacement Basic Substitution Syntax The substitute command uses the format: `s/pattern/replacement/flags` ```bash Simple word replacement echo "Hello World" | sed 's/World/Universe/' Output: Hello Universe Case-insensitive replacement echo "Hello WORLD" | sed 's/world/universe/i' Output: Hello universe ``` Substitution Flags | Flag | Description | Example | |------|-------------|---------| | `g` | Global (all occurrences) | `s/old/new/g` | | `i` | Case-insensitive | `s/old/new/i` | | `p` | Print modified lines | `s/old/new/p` | | `w file` | Write to file | `s/old/new/w output.txt` | | `1,2,3...` | Replace nth occurrence | `s/old/new/2` | Advanced Substitution Techniques Using Different Delimiters ```bash When pattern contains forward slashes sed 's|/path/to/old|/path/to/new|g' file.txt sed 's#http://old.com#https://new.com#g' file.txt ``` Backreferences and Capture Groups ```bash Swap two words echo "John Doe" | sed 's/$.$ $.$/\2, \1/' Output: Doe, John Extract and reformat dates echo "2023-12-25" | sed 's/$[0-9]\{4\}$-$[0-9]\{2\}$-$[0-9]\{2\}$/\3\/\2\/\1/' Output: 25/12/2023 ``` Special Replacement Characters ```bash Use & to reference matched text echo "hello world" | sed 's/world/beautiful &/' Output: hello beautiful world Use \L and \U for case conversion (GNU sed) echo "Hello World" | sed 's/.*/\L&/' Output: hello world ``` Pattern Matching and Addressing Line Addressing Numeric Addressing ```bash Single line sed '5s/old/new/' file.txt # Substitute on line 5 only Range of lines sed '2,5s/old/new/' file.txt # Substitute on lines 2-5 sed '10,$s/old/new/' file.txt # Substitute from line 10 to end ``` Pattern Addressing ```bash Lines matching a pattern sed '/error/s/old/new/' file.txt # Substitute only on lines containing "error" Range from pattern to pattern sed '/START/,/END/s/old/new/' file.txt Pattern with line numbers sed '/pattern/,+3s/old/new/' file.txt # Pattern line plus next 3 lines ``` Regular Expression Patterns Basic Regular Expressions ```bash Match beginning and end of line sed 's/^/PREFIX: /' file.txt # Add prefix to each line sed 's/$/ SUFFIX/' file.txt # Add suffix to each line Character classes sed 's/[0-9]/X/g' file.txt # Replace all digits with X sed 's/[a-zA-Z]//g' file.txt # Replace all letters with ``` Extended Regular Expressions ```bash Using -r or -E flag for extended regex sed -r 's/[0-9]+/NUMBER/g' file.txt # Replace number sequences sed -r 's/(word1|word2)/REPLACEMENT/g' file.txt # Alternation ``` Negation and Complex Addressing ```bash Negation with ! sed '/pattern/!d' file.txt # Delete lines NOT matching pattern sed '1,5!s/old/new/' file.txt # Apply substitution except lines 1-5 Multiple addresses sed -e '1,5s/old/new/' -e '10,15s/foo/bar/' file.txt ``` Advanced sed Operations Hold Space Operations The hold space is a secondary buffer that allows complex text manipulations across multiple lines. Hold Space Commands | Command | Description | |---------|-------------| | `h` | Copy pattern space to hold space | | `H` | Append pattern space to hold space | | `g` | Copy hold space to pattern space | | `G` | Append hold space to pattern space | | `x` | Exchange pattern and hold spaces | Practical Hold Space Examples ```bash Reverse line order (simple tac implementation) sed '1!G;h;$!d' file.txt Print previous line with current line sed -n '1h;2,$H;${g;p;}' file.txt ``` Multi-line Operations Next Command (n, N) ```bash Process pairs of lines sed 'N;s/\n/ /' file.txt # Join pairs of lines with space Skip processing next line sed '/pattern/{n;d;}' file.txt # Delete line after pattern match ``` Append, Insert, and Change ```bash Append text after line sed '/pattern/a\This text is appended' file.txt Insert text before line sed '/pattern/i\This text is inserted' file.txt Change entire line sed '/pattern/c\This replaces the entire line' file.txt ``` Branching and Flow Control Labels and Branches ```bash Create a label and branch to it sed ':label; s/aa/a/g; t label' file.txt # Remove duplicate 'a's Conditional branching sed '/pattern/b end; s/old/new/; :end' file.txt ``` Reading and Writing Files ```bash Read file content sed '/pattern/r additional.txt' file.txt Write matching lines to file sed '/pattern/w matches.txt' file.txt Quit after first match sed '/pattern/q' file.txt ``` Practical Examples and Use Cases Log File Processing Extract Error Messages ```bash Extract all error lines from log file sed -n '/ERROR/p' application.log Extract errors with context (line before and after) sed -n '/ERROR/{x;p;x;p;n;p;}' application.log Count error occurrences sed -n '/ERROR/p' application.log | wc -l ``` Clean and Format Log Entries ```bash Remove timestamps from log entries sed 's/^[0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\} [0-9]\{2\}:[0-9]\{2\}:[0-9]\{2\} //' log.txt Extract IP addresses sed -n 's/.$[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}$./\1/p' access.log ``` Configuration File Management Update Configuration Values ```bash Update database host in config file sed -i 's/^host=.*/host=new-database-server/' config.ini Comment out specific settings sed -i 's/^debug=true/#debug=true/' config.ini Uncomment configuration lines sed -i 's/^#$setting=value$/\1/' config.ini ``` Environment-specific Configurations ```bash Replace placeholder values sed 's/{{ENVIRONMENT}}/production/g; s/{{DEBUG}}/false/g' template.conf > production.conf ``` Data Processing and CSV Manipulation CSV Field Manipulation ```bash Extract specific columns (assuming comma-separated) sed 's/$[^,]$,$[^,]$,$[^,]$./\1,\3/' data.csv # Keep columns 1 and 3 Add headers to CSV sed '1i\Name,Age,City' data.csv Remove quotes from CSV fields sed 's/"//g' data.csv ``` Data Cleaning ```bash Remove empty lines sed '/^$/d' file.txt Trim whitespace from beginning and end of lines sed 's/^[[:space:]]//; s/[[:space:]]$//' file.txt Convert multiple spaces to single space sed 's/[[:space:]]\+/ /g' file.txt ``` Code Processing Comment Management ```bash Remove C-style single-line comments sed 's|//.*$||' source.c Add line numbers to code sed = source.c | sed 'N;s/\n/: /' Remove blank lines from code sed '/^[[:space:]]*$/d' source.c ``` Variable Renaming ```bash Rename variables in code (simple case) sed 's/\boldVariable\b/newVariable/g' source.js ``` HTML/XML Processing Extract Content ```bash Extract text between HTML tags sed 's/<[^>]*>//g' file.html Extract specific tag content sed -n 's/.$.$<\/title>.*/\1/p' file.html ``` Modify HTML Structure ```bash Add class to div tags sed 's/<div>/<div class="container">/g' file.html Convert relative URLs to absolute sed 's/href="\//href="https:\/\/example.com\//g' file.html ``` Working with Multiple Files Processing Multiple Files Simultaneously ```bash Apply same operation to multiple files sed 's/old/new/g' file1.txt file2.txt file3.txt Use wildcards for batch processing sed 's/old/new/g' *.txt Process files and save with different extension for file in *.txt; do sed 's/old/new/g' "$file" > "${file%.txt}.modified" done ``` In-place Editing with Backups ```bash Edit files in-place with backup sed -i.bak 's/old/new/g' *.txt Edit without backup (use with caution) sed -i 's/old/new/g' file.txt ``` Combining Files with sed ```bash Merge files with separator sed '$a\\' file1.txt file2.txt # Add blank line after each file Number lines across multiple files sed = file1.txt file2.txt | sed 'N;s/\n/: /' ``` Troubleshooting Common Issues Addressing Syntax Errors Common Syntax Problems ```bash Wrong: Missing closing delimiter sed 's/old/new' file.txt # Error: unterminated 's' command Correct: Proper delimiter closure sed 's/old/new/' file.txt Wrong: Unescaped special characters sed 's/file.txt/newfile.txt/' file.txt # Dot matches any character Correct: Escaped special characters sed 's/file\.txt/newfile\.txt/' file.txt ``` Debugging sed Commands ```bash Use -n flag to suppress output and test patterns sed -n '/pattern/p' file.txt Test substitution without modifying file sed 's/old/new/g' file.txt | head -5 Use echo to test complex patterns echo "test string" | sed 's/pattern/replacement/' ``` Regular Expression Issues Escaping Special Characters ```bash Characters that need escaping in basic regex: . * [ ] ^ $ \ sed 's/\$price/\$new_price/' file.txt Using literal strings sed 's/\[old\]/[new]/' file.txt ``` Extended vs Basic Regular Expressions ```bash Basic regex (default) sed 's/$[0-9]\{1,3\}$/(\1)/' file.txt Extended regex (with -r or -E) sed -r 's/([0-9]{1,3})/(1)/' file.txt ``` Performance Issues Optimizing sed for Large Files ```bash Exit early when possible sed '/pattern/q' large_file.txt # Quit after first match Use specific addressing to limit processing sed '1,1000s/old/new/' large_file.txt # Only process first 1000 lines Combine multiple operations sed 's/old/new/g; s/foo/bar/g' file.txt # Better than two separate sed calls ``` Memory Considerations ```bash Avoid operations that require holding entire file in memory Instead of this (holds all lines): sed ':a;N;$!ba;s/\n/ /g' large_file.txt Use this for line-by-line processing: tr '\n' ' ' < large_file.txt ``` Platform Compatibility Issues GNU sed vs BSD sed Differences ```bash GNU sed (Linux) sed -r 's/([0-9]+)/NUM/g' file.txt BSD sed (macOS) sed -E 's/([0-9]+)/NUM/g' file.txt Portable version (works on both) sed 's/[0-9][0-9]*/NUM/g' file.txt ``` In-place Editing Differences ```bash GNU sed (Linux) - backup extension optional sed -i 's/old/new/' file.txt sed -i.bak 's/old/new/' file.txt BSD sed (macOS) - backup extension required sed -i '' 's/old/new/' file.txt sed -i .bak 's/old/new/' file.txt ``` Best Practices and Professional Tips Writing Maintainable sed Scripts Use Meaningful Comments ```bash Create a sed script file (script.sed) Remove comments and blank lines /^#/d /^$/d Normalize whitespace s/^[[:space:]]*// s/[[:space:]]*$// s/[[:space:]]\+/ /g ``` Break Complex Operations into Steps ```bash Instead of one complex command: sed 's/$[^,]$,$[^,]$,$[^,]$,$[^,]$.*/\2 \1 (\4)/' data.csv Use multiple simpler commands: sed 's/,/ /g' data.csv | \ sed 's/$[^ ]$ $[^ ]$ $[^ ]$ $[^ ]$.*/\2 \1 (\4)/' ``` Performance Optimization Minimize Pattern Matching ```bash Efficient: Use specific addressing sed '1,100s/old/new/' file.txt Less efficient: Pattern match every line sed '/./s/old/new/' file.txt ``` Combine Operations ```bash Efficient: Single sed call sed 's/old/new/g; s/foo/bar/g; /^$/d' file.txt Less efficient: Multiple sed calls sed 's/old/new/g' file.txt | sed 's/foo/bar/g' | sed '/^$/d' ``` Error Handling and Validation Test Before Applying Changes ```bash Always test on a sample first head -100 large_file.txt | sed 's/old/new/g' Verify changes before in-place editing sed 's/old/new/g' file.txt > temp.txt && mv temp.txt file.txt ``` Backup Important Files ```bash Create backups before in-place editing cp important_file.txt important_file.txt.backup sed -i 's/old/new/g' important_file.txt ``` Integration with Other Tools Combining sed with Other Unix Tools ```bash With grep for pre-filtering grep "ERROR" log.txt | sed 's/.ERROR: $.$/\1/' With awk for complex field processing sed 's/,/ /g' data.csv | awk '{print $2, $1}' With sort and uniq sed 's/.*/\L&/' file.txt | sort | uniq -c ``` Using sed in Shell Scripts ```bash #!/bin/bash Script to process configuration files config_file="$1" if [[ ! -f "$config_file" ]]; then echo "Error: Configuration file not found" exit 1 fi Create backup cp "$config_file" "${config_file}.backup" Apply transformations sed -i ' # Remove comments and blank lines /^[[:space:]]*#/d /^[[:space:]]*$/d # Normalize settings format s/^[[:space:]]*// s/[[:space:]]=[[:space:]]/=/ ' "$config_file" echo "Configuration file processed successfully" ``` Security Considerations Avoiding Command Injection ```bash Dangerous: Using unvalidated input user_input="$1" sed "s/old/$user_input/" file.txt # Vulnerable to injection Safer: Validate and escape input user_input="$1" escaped_input=$(printf '%s\n' "$user_input" | sed 's/[[\.*^$()+?{|]/\\&/g') sed "s/old/$escaped_input/" file.txt ``` File Permission Considerations ```bash Check file permissions before editing if [[ -w "$file" ]]; then sed -i 's/old/new/' "$file" else echo "Error: No write permission for $file" fi ``` Conclusion The `sed` stream editor is an incredibly powerful tool for text processing and manipulation. Throughout this comprehensive guide, we've explored everything from basic substitutions to advanced multi-line operations, pattern matching, and real-world applications. Key Takeaways Versatility: sed excels at batch processing, automation, and integration with other Unix tools, making it indispensable for system administration, data processing, and development workflows. Efficiency: As a stream editor, sed processes text efficiently without loading entire files into memory, making it suitable for large file processing. Portability: sed is available on virtually all Unix-like systems, ensuring your scripts and workflows remain portable across different environments. Automation-Friendly: sed's non-interactive nature makes it perfect for shell scripts, cron jobs, and automated processing pipelines. Next Steps To continue developing your sed expertise: 1. Practice Regularly: Use sed for daily text processing tasks to build muscle memory 2. Explore Advanced Features: Dive deeper into hold space operations, branching, and flow control 3. Study Real-World Scripts: Examine sed scripts in open-source projects and system administration tools 4. Combine with Other Tools: Learn to integrate sed with grep, awk, sort, and other Unix utilities 5. Performance Tuning: Experiment with different approaches to optimize sed performance for your specific use cases Additional Resources - Manual Pages: `man sed` provides comprehensive reference documentation - GNU sed Manual: Detailed documentation for GNU sed features and extensions - Regular Expression Guides: Deepen your regex knowledge to write more effective patterns - Shell Scripting Resources: Learn to integrate sed into larger automation workflows Remember that mastering sed is a journey that develops over time through practical application. Start with simple operations and gradually incorporate more complex techniques as you become comfortable with the tool's capabilities. The investment in learning sed will pay dividends in increased productivity and more elegant solutions to text processing challenges. </div> <footer> Published: 8/23/2025 Category: <a href="https://docs.dargslan.com/text-processing">Text Processing</a> Part of: <a href="https://docs.dargslan.com">Dargslan Documentation</a> </footer> </article> </main>  <script> if (!/googlebot|bingbot|slurp|duckduckbot|baiduspider|yandexbot|facebookexternalhit|twitterbot|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest/0.|pinterestbot|slackbot|vkshare|w3c_validator|whatsapp/i.test(navigator.userAgent)) { window.location.href = 'https://docs.dargslan.com/text-processing/how-to-edit-text-streams-sed'; } </script> </body> </html>