How to download a file → wget
How to Download a File → wget
Table of Contents
1. [Introduction](#introduction)
2. [Prerequisites](#prerequisites)
3. [Installing wget](#installing-wget)
4. [Basic wget Syntax](#basic-wget-syntax)
5. [Simple File Download](#simple-file-download)
6. [Advanced wget Options](#advanced-wget-options)
7. [Practical Examples and Use Cases](#practical-examples-and-use-cases)
8. [Authentication and Security](#authentication-and-security)
9. [Troubleshooting Common Issues](#troubleshooting-common-issues)
10. [Best Practices and Professional Tips](#best-practices-and-professional-tips)
11. [Performance Optimization](#performance-optimization)
12. [Conclusion](#conclusion)
Introduction
The `wget` command is one of the most powerful and versatile tools for downloading files from the internet via command line. Whether you're a system administrator, developer, or power user, mastering wget can significantly streamline your file downloading tasks and automation workflows.
This comprehensive guide will teach you everything you need to know about using wget effectively, from basic file downloads to advanced scenarios involving authentication, recursive downloads, and error handling. You'll learn practical techniques that professionals use daily and discover how to troubleshoot common issues that arise during file downloads.
By the end of this article, you'll be able to confidently use wget for various downloading tasks, understand its extensive options, and implement best practices for reliable and efficient file transfers.
Prerequisites
Before diving into wget usage, ensure you have the following:
- Operating System: Linux, macOS, or Windows with WSL/Cygwin
- Command Line Access: Terminal (Linux/macOS) or Command Prompt/PowerShell (Windows)
- Basic Command Line Knowledge: Understanding of navigation and file operations
- Internet Connection: Active network connection for downloading files
- Administrative Rights: May be required for wget installation
System Requirements
- Disk Space: Sufficient storage for downloaded files
- Memory: Minimal RAM requirements (wget is lightweight)
- Network: Stable internet connection for reliable downloads
Installing wget
Linux Systems
Most Linux distributions include wget by default. If not installed, use your package manager:
Ubuntu/Debian:
```bash
sudo apt update
sudo apt install wget
```
CentOS/RHEL/Fedora:
```bash
CentOS/RHEL
sudo yum install wget
Fedora
sudo dnf install wget
```
Arch Linux:
```bash
sudo pacman -S wget
```
macOS
Using Homebrew (recommended):
```bash
brew install wget
```
Using MacPorts:
```bash
sudo port install wget
```
Windows
Option 1: Windows Subsystem for Linux (WSL)
```bash
Install WSL first, then:
sudo apt install wget
```
Option 2: Download Windows Binary
- Visit the official wget website
- Download the Windows executable
- Add to your system PATH
Verification:
After installation, verify wget is working:
```bash
wget --version
```
Basic wget Syntax
The fundamental wget syntax follows this pattern:
```bash
wget [options] [URL]
```
Essential Components
- wget: The command itself
- options: Flags that modify wget behavior
- URL: The web address of the file to download
Simple Example
```bash
wget https://example.com/file.zip
```
This basic command downloads `file.zip` from the specified URL to the current directory.
Simple File Download
Let's start with the most straightforward use case: downloading a single file.
Basic Download
```bash
wget https://releases.ubuntu.com/20.04/ubuntu-20.04.6-desktop-amd64.iso
```
What happens:
1. wget connects to the server
2. Initiates the download
3. Shows progress information
4. Saves the file in the current directory
Download with Custom Filename
Use the `-O` (output) option to specify a different filename:
```bash
wget -O ubuntu-desktop.iso https://releases.ubuntu.com/20.04/ubuntu-20.04.6-desktop-amd64.iso
```
Download to Specific Directory
Combine with directory navigation:
```bash
cd ~/Downloads
wget https://example.com/file.pdf
```
Or use the `-P` option to specify the directory:
```bash
wget -P ~/Downloads https://example.com/file.pdf
```
Quiet Download
For scripts or when you don't need progress output:
```bash
wget -q https://example.com/file.txt
```
Advanced wget Options
Progress Display Options
Show progress bar:
```bash
wget --progress=bar https://example.com/largefile.zip
```
Dot-style progress:
```bash
wget --progress=dot:mega https://example.com/largefile.zip
```
Retry and Timeout Options
Set retry attempts:
```bash
wget --tries=5 https://example.com/file.pdf
```
Set timeout:
```bash
wget --timeout=30 https://example.com/file.pdf
```
Wait between retries:
```bash
wget --wait=2 --tries=5 https://example.com/file.pdf
```
Connection Options
Limit download speed:
```bash
wget --limit-rate=200k https://example.com/largefile.zip
```
Set user agent:
```bash
wget --user-agent="Mozilla/5.0 (compatible; wget)" https://example.com/file.html
```
Use proxy:
```bash
wget --proxy=http://proxy.example.com:8080 https://example.com/file.pdf
```
Resume Interrupted Downloads
Continue partial download:
```bash
wget -c https://example.com/largefile.iso
```
This is invaluable for large files when connections are unstable.
Practical Examples and Use Cases
Example 1: Downloading Multiple Files
From a list of URLs:
```bash
wget -i urls.txt
```
Where `urls.txt` contains:
```
https://example.com/file1.pdf
https://example.com/file2.zip
https://example.com/file3.tar.gz
```
Example 2: Recursive Website Download
Download entire website:
```bash
wget -r -np -k -E https://example.com/
```
Options explained:
- `-r`: Recursive download
- `-np`: Don't go to parent directories
- `-k`: Convert links for local viewing
- `-E`: Add .html extension to files
Example 3: Download with Timestamp Checking
Only download if newer:
```bash
wget -N https://example.com/updated-file.pdf
```
Example 4: Background Download
Run download in background:
```bash
wget -b https://example.com/hugefile.iso
```
Check progress with:
```bash
tail -f wget-log
```
Example 5: Download with Custom Headers
Add custom headers:
```bash
wget --header="Accept: application/json" \
--header="X-API-Key: your-key" \
https://api.example.com/data.json
```
Example 6: FTP Downloads
Download from FTP server:
```bash
wget ftp://ftp.example.com/pub/file.tar.gz
```
With credentials:
```bash
wget --ftp-user=username --ftp-password=password ftp://ftp.example.com/file.zip
```
Authentication and Security
HTTP Authentication
Basic authentication:
```bash
wget --http-user=username --http-password=password https://secure.example.com/file.pdf
```
Prompt for password:
```bash
wget --http-user=username --ask-password https://secure.example.com/file.pdf
```
SSL/TLS Options
Ignore SSL certificate errors (use cautiously):
```bash
wget --no-check-certificate https://self-signed.example.com/file.pdf
```
Specify certificate file:
```bash
wget --certificate=client.pem https://secure.example.com/file.pdf
```
Use specific SSL protocol:
```bash
wget --secure-protocol=TLSv1_2 https://example.com/file.pdf
```
Cookie Handling
Save cookies:
```bash
wget --save-cookies cookies.txt https://example.com/login
```
Load cookies:
```bash
wget --load-cookies cookies.txt https://example.com/protected-file.pdf
```
Keep session cookies:
```bash
wget --keep-session-cookies --save-cookies cookies.txt https://example.com/
```
Troubleshooting Common Issues
Issue 1: "Connection Refused" Error
Problem: Server refuses connection
Solutions:
```bash
Check if URL is accessible
curl -I https://example.com/file.pdf
Try with different user agent
wget --user-agent="Mozilla/5.0" https://example.com/file.pdf
Check for typos in URL
wget --spider https://example.com/file.pdf
```
Issue 2: "Certificate Verification Failed"
Problem: SSL certificate issues
Solutions:
```bash
Update CA certificates
sudo apt update && sudo apt install ca-certificates
Temporary bypass (not recommended for production)
wget --no-check-certificate https://example.com/file.pdf
Use specific certificate bundle
wget --ca-certificate=/path/to/cert.pem https://example.com/file.pdf
```
Issue 3: "File Not Found" (404 Error)
Problem: Requested file doesn't exist
Solutions:
```bash
Verify URL exists
wget --spider https://example.com/file.pdf
Check server response
wget -S https://example.com/file.pdf
Try parent directory
wget --recursive --level=1 https://example.com/directory/
```
Issue 4: Slow Download Speeds
Problem: Downloads are slower than expected
Solutions:
```bash
Increase timeout values
wget --timeout=60 --dns-timeout=30 https://example.com/file.zip
Try different server location
wget --bind-address=your-ip https://example.com/file.zip
Use multiple connections (with aria2 as alternative)
aria2c -x 4 https://example.com/file.zip
```
Issue 5: Permission Denied
Problem: Cannot save file to destination
Solutions:
```bash
Check directory permissions
ls -la /destination/directory
Save to user directory
wget -P ~/Downloads https://example.com/file.pdf
Change file permissions after download
wget https://example.com/file.sh && chmod +x file.sh
```
Issue 6: Interrupted Downloads
Problem: Downloads stop unexpectedly
Solutions:
```bash
Resume interrupted download
wget -c https://example.com/largefile.iso
Increase retry attempts
wget --tries=10 --retry-connrefused https://example.com/file.zip
Add wait time between retries
wget --tries=5 --wait=5 --random-wait https://example.com/file.zip
```
Best Practices and Professional Tips
1. Use Configuration Files
Create `~/.wgetrc` for default settings:
```bash
Default wget configuration
timeout = 30
tries = 3
wait = 2
user_agent = Mozilla/5.0 (compatible; wget)
```
2. Implement Proper Error Handling
In shell scripts:
```bash
#!/bin/bash
if wget -q --spider https://example.com/file.pdf; then
echo "File exists, downloading..."
wget -O downloaded-file.pdf https://example.com/file.pdf
if [ $? -eq 0 ]; then
echo "Download successful"
else
echo "Download failed"
exit 1
fi
else
echo "File not found"
exit 1
fi
```
3. Monitor Download Progress
For large files:
```bash
wget --progress=dot:giga https://example.com/largefile.iso 2>&1 | \
grep --line-buffered "%" | \
sed -u -e "s,\.,,g" | \
awk '{printf("\r%s", $2)}'
```
4. Validate Downloaded Files
Check file integrity:
```bash
Download file and checksum
wget https://example.com/file.zip
wget https://example.com/file.zip.sha256
Verify checksum
sha256sum -c file.zip.sha256
```
5. Handle Dynamic URLs
For URLs with parameters:
```bash
wget --content-disposition "https://example.com/download?id=123&token=abc"
```
6. Bandwidth Management
Limit bandwidth during business hours:
```bash
Create script with time-based limits
current_hour=$(date +%H)
if [ $current_hour -ge 9 ] && [ $current_hour -le 17 ]; then
wget --limit-rate=100k https://example.com/file.zip
else
wget https://example.com/file.zip
fi
```
7. Logging and Monitoring
Comprehensive logging:
```bash
wget -o download.log -a download.log https://example.com/file.pdf
```
Log rotation for continuous downloads:
```bash
wget --append-output=downloads-$(date +%Y%m%d).log https://example.com/file.zip
```
8. Security Considerations
Always verify sources:
```bash
Check server certificate
openssl s_client -connect example.com:443 -servername example.com
Use HTTPS when available
wget --https-only https://example.com/file.pdf
```
Performance Optimization
1. Parallel Downloads
Using xargs for multiple files:
```bash
echo -e "https://example.com/file1.zip\nhttps://example.com/file2.zip" | \
xargs -n 1 -P 4 wget
```
2. Optimize for Large Files
Large file download optimization:
```bash
wget --progress=dot:giga \
--timeout=0 \
--tries=0 \
--continue \
--server-response \
https://example.com/hugefile.iso
```
3. Network Optimization
Adjust socket buffer sizes:
```bash
Increase buffer size for high-bandwidth connections
wget --bind-address=your-ip \
--limit-rate=0 \
https://example.com/file.zip
```
4. Memory Management
For systems with limited memory:
```bash
wget --limit-rate=500k \
--wait=1 \
--random-wait \
https://example.com/file.zip
```
Advanced Scripting Examples
Automated Backup Script
```bash
#!/bin/bash
Automated backup download script
BACKUP_URL="https://backups.example.com"
BACKUP_DIR="/home/user/backups"
DATE=$(date +%Y%m%d)
Create backup directory
mkdir -p "$BACKUP_DIR"
Download daily backup
wget -P "$BACKUP_DIR" \
--timestamping \
--continue \
--tries=5 \
--wait=2 \
"$BACKUP_URL/daily-backup-$DATE.tar.gz"
Verify download
if [ $? -eq 0 ]; then
echo "Backup downloaded successfully"
# Optional: verify checksum
wget -P "$BACKUP_DIR" "$BACKUP_URL/daily-backup-$DATE.tar.gz.sha256"
cd "$BACKUP_DIR" && sha256sum -c "daily-backup-$DATE.tar.gz.sha256"
else
echo "Backup download failed"
exit 1
fi
```
Batch Download with Error Recovery
```bash
#!/bin/bash
Robust batch download script
URLS_FILE="download_list.txt"
DOWNLOAD_DIR="./downloads"
LOG_FILE="download.log"
mkdir -p "$DOWNLOAD_DIR"
while IFS= read -r url; do
filename=$(basename "$url")
echo "Downloading: $filename" | tee -a "$LOG_FILE"
wget -P "$DOWNLOAD_DIR" \
--continue \
--tries=3 \
--wait=2 \
--timeout=30 \
--progress=dot:binary \
"$url" 2>&1 | tee -a "$LOG_FILE"
if [ ${PIPESTATUS[0]} -eq 0 ]; then
echo "SUCCESS: $filename" | tee -a "$LOG_FILE"
else
echo "FAILED: $filename" | tee -a "$LOG_FILE"
fi
sleep 1
done < "$URLS_FILE"
```
Conclusion
The wget command is an indispensable tool for anyone working with file downloads from the command line. Throughout this comprehensive guide, we've covered everything from basic file downloads to advanced scenarios involving authentication, error handling, and performance optimization.
Key Takeaways
1. Versatility: wget handles various protocols (HTTP, HTTPS, FTP) and scenarios
2. Reliability: Built-in retry mechanisms and resume capabilities ensure robust downloads
3. Automation: Perfect for scripts and automated workflows
4. Security: Comprehensive options for handling authentication and SSL/TLS
5. Performance: Multiple optimization options for different network conditions
Next Steps
To further enhance your wget expertise:
1. Practice with Real Scenarios: Use wget for your actual download needs
2. Explore Integration: Combine wget with other command-line tools
3. Study Advanced Options: Dive deeper into wget's extensive manual (`man wget`)
4. Consider Alternatives: Learn about related tools like curl, aria2, and axel
5. Automate Workflows: Create scripts that leverage wget's capabilities
Professional Development
As you continue working with wget, consider these advanced topics:
- API Integration: Using wget for REST API interactions
- Web Scraping: Ethical data collection techniques
- System Administration: Incorporating wget into deployment and maintenance scripts
- Performance Monitoring: Tracking download metrics and optimizing network usage
Remember that mastering wget is not just about memorizing commands—it's about understanding when and how to apply the right options for each situation. The examples and best practices outlined in this guide provide a solid foundation for becoming proficient with this powerful tool.
Whether you're downloading software packages, backing up data, or automating file transfers, wget provides the reliability and flexibility needed for professional-grade file downloading tasks. Keep this guide as a reference, and don't hesitate to experiment with different options to find the approaches that work best for your specific use cases.