How to configure Kubernetes secrets in Linux
How to Configure Kubernetes Secrets in Linux
Kubernetes secrets are essential components for managing sensitive data in containerized applications. This comprehensive guide will walk you through everything you need to know about configuring, creating, and managing Kubernetes secrets in Linux environments. Whether you're a beginner just starting with Kubernetes or an experienced developer looking to enhance your security practices, this article provides detailed instructions and practical examples to help you master Kubernetes secrets management.
Table of Contents
1. [Introduction to Kubernetes Secrets](#introduction-to-kubernetes-secrets)
2. [Prerequisites and Requirements](#prerequisites-and-requirements)
3. [Understanding Secret Types](#understanding-secret-types)
4. [Creating Kubernetes Secrets](#creating-kubernetes-secrets)
5. [Managing Secrets with kubectl](#managing-secrets-with-kubectl)
6. [Using Secrets in Pods and Deployments](#using-secrets-in-pods-and-deployments)
7. [Advanced Configuration Techniques](#advanced-configuration-techniques)
8. [Security Best Practices](#security-best-practices)
9. [Troubleshooting Common Issues](#troubleshooting-common-issues)
10. [Monitoring and Maintenance](#monitoring-and-maintenance)
11. [Conclusion and Next Steps](#conclusion-and-next-steps)
Introduction to Kubernetes Secrets
Kubernetes secrets provide a secure way to store and manage sensitive information such as passwords, OAuth tokens, SSH keys, and other confidential data within your cluster. Unlike ConfigMaps, which are designed for non-confidential configuration data, secrets are specifically designed to hold sensitive information and provide additional security features.
Secrets are stored in etcd in base64-encoded format and can be mounted as files in pods or used as environment variables. They offer several advantages over hardcoding sensitive data directly into container images or pod specifications, including improved security, easier credential rotation, and better separation of concerns.
Key Benefits of Kubernetes Secrets
- Enhanced Security: Secrets are base64-encoded and can be encrypted at rest
- Flexible Usage: Can be mounted as volumes or exposed as environment variables
- Centralized Management: Single source of truth for sensitive configuration data
- Access Control: Integration with Kubernetes RBAC for fine-grained permissions
- Automatic Updates: Secrets can be updated without rebuilding container images
Prerequisites and Requirements
Before diving into Kubernetes secrets configuration, ensure you have the following prerequisites in place:
System Requirements
- Linux Distribution: Ubuntu 18.04+, CentOS 7+, or RHEL 7+
- Kubernetes Cluster: Version 1.18 or higher
- kubectl: Command-line tool configured to communicate with your cluster
- Docker: Container runtime (if using Docker as the container runtime)
- Root or sudo access: For system-level configurations
Software Installation
First, verify your Kubernetes installation:
```bash
kubectl version --client
kubectl cluster-info
```
If you need to install kubectl on Linux:
```bash
Download the latest kubectl binary
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
Make it executable
chmod +x kubectl
Move to PATH
sudo mv kubectl /usr/local/bin/
Verify installation
kubectl version --client
```
Cluster Access Verification
Ensure you can access your Kubernetes cluster:
```bash
kubectl get nodes
kubectl get namespaces
```
Understanding Secret Types
Kubernetes supports several types of secrets, each designed for specific use cases. Understanding these types is crucial for proper secret management.
Generic Secrets (Opaque)
The most common type of secret, used for arbitrary user-defined data:
```bash
kubectl create secret generic my-secret --from-literal=username=admin --from-literal=password=secretpassword
```
Docker Registry Secrets
Used for authenticating with private Docker registries:
```bash
kubectl create secret docker-registry regcred \
--docker-server=https://index.docker.io/v1/ \
--docker-username=myusername \
--docker-password=mypassword \
--docker-email=myemail@example.com
```
TLS Secrets
Specifically designed for storing TLS certificates and keys:
```bash
kubectl create secret tls tls-secret \
--cert=path/to/tls.crt \
--key=path/to/tls.key
```
Service Account Token Secrets
Used for service account authentication (automatically managed by Kubernetes):
```yaml
apiVersion: v1
kind: Secret
metadata:
name: sa-token-secret
annotations:
kubernetes.io/service-account.name: my-service-account
type: kubernetes.io/service-account-token
```
Creating Kubernetes Secrets
There are multiple ways to create Kubernetes secrets. Let's explore each method with practical examples.
Method 1: Using kubectl create secret
The simplest way to create secrets is using the kubectl command-line tool.
Creating from Literal Values
```bash
Create a generic secret with literal values
kubectl create secret generic database-credentials \
--from-literal=username=dbuser \
--from-literal=password=dbpass123 \
--from-literal=database=myapp
Verify the secret creation
kubectl get secrets
kubectl describe secret database-credentials
```
Creating from Files
First, create files containing your sensitive data:
```bash
Create credential files
echo -n 'admin' > username.txt
echo -n 'supersecretpassword' > password.txt
Create secret from files
kubectl create secret generic file-credentials \
--from-file=username.txt \
--from-file=password.txt
Clean up files
rm username.txt password.txt
```
Creating from Environment File
Create an environment file:
```bash
Create .env file
cat << EOF > database.env
DB_HOST=mysql.example.com
DB_PORT=3306
DB_NAME=production
DB_USER=appuser
DB_PASSWORD=complexpassword123
EOF
Create secret from env file
kubectl create secret generic database-config --from-env-file=database.env
Clean up
rm database.env
```
Method 2: Using YAML Manifests
For more complex configurations and version control, use YAML manifests.
Basic Secret Manifest
```yaml
secret-basic.yaml
apiVersion: v1
kind: Secret
metadata:
name: basic-auth
namespace: default
type: Opaque
data:
username: YWRtaW4= # base64 encoded 'admin'
password: cGFzc3dvcmQxMjM= # base64 encoded 'password123'
```
Apply the secret:
```bash
kubectl apply -f secret-basic.yaml
```
String Data for Easier Management
Use `stringData` to avoid manual base64 encoding:
```yaml
secret-stringdata.yaml
apiVersion: v1
kind: Secret
metadata:
name: app-config
namespace: default
type: Opaque
stringData:
api-key: "your-api-key-here"
database-url: "postgresql://user:pass@localhost:5432/mydb"
jwt-secret: "your-jwt-secret-key"
```
```bash
kubectl apply -f secret-stringdata.yaml
```
Method 3: Creating TLS Secrets
For HTTPS applications, create TLS secrets:
```bash
Generate a private key
openssl genrsa -out tls.key 2048
Generate a certificate signing request
openssl req -new -key tls.key -out tls.csr -subj "/CN=myapp.example.com"
Generate a self-signed certificate (for testing)
openssl x509 -req -in tls.csr -signkey tls.key -out tls.crt -days 365
Create TLS secret
kubectl create secret tls myapp-tls \
--cert=tls.crt \
--key=tls.key
Clean up files
rm tls.key tls.csr tls.crt
```
Managing Secrets with kubectl
Effective secret management requires understanding various kubectl operations.
Viewing Secrets
```bash
List all secrets in current namespace
kubectl get secrets
List secrets in all namespaces
kubectl get secrets --all-namespaces
Get detailed information about a secret
kubectl describe secret database-credentials
View secret data (base64 encoded)
kubectl get secret database-credentials -o yaml
Decode secret values
kubectl get secret database-credentials -o jsonpath='{.data.username}' | base64 --decode
kubectl get secret database-credentials -o jsonpath='{.data.password}' | base64 --decode
```
Updating Secrets
Method 1: Using kubectl patch
```bash
Update a secret value
kubectl patch secret database-credentials -p='{"data":{"password":"bmV3cGFzc3dvcmQ="}}'
Update using stringData
kubectl patch secret database-credentials -p='{"stringData":{"password":"newpassword"}}'
```
Method 2: Using kubectl edit
```bash
kubectl edit secret database-credentials
```
Method 3: Replace with new manifest
```bash
Update the YAML file and reapply
kubectl apply -f secret-basic.yaml
```
Deleting Secrets
```bash
Delete a specific secret
kubectl delete secret database-credentials
Delete multiple secrets
kubectl delete secret secret1 secret2 secret3
Delete all secrets in a namespace (use with caution)
kubectl delete secrets --all
```
Using Secrets in Pods and Deployments
Secrets become useful when consumed by applications running in pods. There are two primary methods to use secrets in pods.
Method 1: Environment Variables
Single Secret as Environment Variables
```yaml
pod-with-secret-env.yaml
apiVersion: v1
kind: Pod
metadata:
name: secret-env-pod
spec:
containers:
- name: app-container
image: nginx:latest
env:
- name: DB_USERNAME
valueFrom:
secretKeyRef:
name: database-credentials
key: username
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: database-credentials
key: password
```
All Secret Keys as Environment Variables
```yaml
pod-with-secret-envfrom.yaml
apiVersion: v1
kind: Pod
metadata:
name: secret-envfrom-pod
spec:
containers:
- name: app-container
image: nginx:latest
envFrom:
- secretRef:
name: database-credentials
```
Method 2: Volume Mounts
Mounting Secret as Files
```yaml
pod-with-secret-volume.yaml
apiVersion: v1
kind: Pod
metadata:
name: secret-volume-pod
spec:
containers:
- name: app-container
image: nginx:latest
volumeMounts:
- name: secret-volume
mountPath: "/etc/secrets"
readOnly: true
volumes:
- name: secret-volume
secret:
secretName: database-credentials
```
Mounting Specific Keys
```yaml
pod-with-secret-selective.yaml
apiVersion: v1
kind: Pod
metadata:
name: secret-selective-pod
spec:
containers:
- name: app-container
image: nginx:latest
volumeMounts:
- name: secret-volume
mountPath: "/etc/secrets"
readOnly: true
volumes:
- name: secret-volume
secret:
secretName: database-credentials
items:
- key: username
path: db-username
- key: password
path: db-password
mode: 0400 # Read-only for owner
```
Using Secrets in Deployments
```yaml
deployment-with-secrets.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: web-app
spec:
replicas: 3
selector:
matchLabels:
app: web-app
template:
metadata:
labels:
app: web-app
spec:
containers:
- name: web-container
image: nginx:latest
ports:
- containerPort: 80
env:
- name: API_KEY
valueFrom:
secretKeyRef:
name: app-config
key: api-key
- name: DB_CONNECTION
valueFrom:
secretKeyRef:
name: app-config
key: database-url
volumeMounts:
- name: tls-certs
mountPath: "/etc/ssl/certs"
readOnly: true
volumes:
- name: tls-certs
secret:
secretName: myapp-tls
```
Apply the deployment:
```bash
kubectl apply -f deployment-with-secrets.yaml
```
Advanced Configuration Techniques
Namespace-Specific Secrets
Create secrets in specific namespaces:
```bash
Create namespace
kubectl create namespace production
Create secret in specific namespace
kubectl create secret generic prod-credentials \
--from-literal=username=produser \
--from-literal=password=prodpass \
--namespace=production
```
Using Secrets with Service Accounts
```yaml
serviceaccount-with-secret.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: app-service-account
secrets:
- name: app-service-account-token
---
apiVersion: v1
kind: Secret
metadata:
name: app-service-account-token
annotations:
kubernetes.io/service-account.name: app-service-account
type: kubernetes.io/service-account-token
```
Immutable Secrets
For enhanced security and performance, create immutable secrets:
```yaml
immutable-secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: immutable-config
type: Opaque
data:
config.json: eyJhcGkiOiJodHRwczovL2FwaS5leGFtcGxlLmNvbSJ9
immutable: true
```
Secret Rotation Strategy
Implement a secret rotation strategy:
```bash
#!/bin/bash
rotate-secret.sh
SECRET_NAME="database-credentials"
NEW_PASSWORD=$(openssl rand -base64 32)
Create new secret with versioned name
kubectl create secret generic "${SECRET_NAME}-v2" \
--from-literal=username=dbuser \
--from-literal=password="${NEW_PASSWORD}"
Update deployment to use new secret
kubectl patch deployment web-app -p='{"spec":{"template":{"spec":{"containers":[{"name":"web-container","env":[{"name":"DB_PASSWORD","valueFrom":{"secretKeyRef":{"name":"database-credentials-v2","key":"password"}}}]}]}}}}'
Wait for rollout to complete
kubectl rollout status deployment/web-app
Delete old secret
kubectl delete secret "${SECRET_NAME}"
Rename new secret
kubectl get secret "${SECRET_NAME}-v2" -o yaml | \
sed "s/${SECRET_NAME}-v2/${SECRET_NAME}/g" | \
kubectl apply -f -
kubectl delete secret "${SECRET_NAME}-v2"
```
Security Best Practices
Encryption at Rest
Enable encryption at rest for etcd:
```yaml
encryption-config.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
providers:
- aescbc:
keys:
- name: key1
secret:
- identity: {}
```
RBAC for Secrets
Create specific RBAC rules for secret access:
```yaml
secret-rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: default
name: secret-reader
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: read-secrets
namespace: default
subjects:
- kind: ServiceAccount
name: app-service-account
namespace: default
roleRef:
kind: Role
name: secret-reader
apiGroup: rbac.authorization.k8s.io
```
Secret Validation
Implement admission controllers to validate secrets:
```yaml
secret-validation-policy.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: require-secret-labels
spec:
validationFailureAction: enforce
background: false
rules:
- name: check-secret-labels
match:
any:
- resources:
kinds:
- Secret
validate:
message: "Secrets must have 'app' and 'environment' labels"
pattern:
metadata:
labels:
app: "?*"
environment: "?*"
```
Least Privilege Access
```bash
Create service account with minimal permissions
kubectl create serviceaccount limited-app-sa
Create role with specific secret access
kubectl create role secret-accessor \
--verb=get,list \
--resource=secrets \
--resource-name=app-config,database-credentials
Bind role to service account
kubectl create rolebinding limited-app-binding \
--role=secret-accessor \
--serviceaccount=default:limited-app-sa
```
Troubleshooting Common Issues
Issue 1: Secret Not Found
Problem: Pod fails to start with "secret not found" error.
Diagnosis:
```bash
kubectl describe pod
kubectl get secrets
kubectl get secrets -n
```
Solution:
```bash
Verify secret exists in correct namespace
kubectl get secret -n
Create missing secret
kubectl create secret generic \
--from-literal=key=value \
--namespace=
```
Issue 2: Base64 Encoding Issues
Problem: Secret values appear corrupted or incorrect.
Diagnosis:
```bash
Check current secret value
kubectl get secret -o yaml
Decode value manually
echo "encoded-value" | base64 --decode
```
Solution:
```bash
Use stringData instead of data
kubectl patch secret -p='{"stringData":{"key":"correct-value"}}'
Or recreate with correct encoding
echo -n "correct-value" | base64
kubectl patch secret -p='{"data":{"key":""}}'
```
Issue 3: Permission Denied
Problem: Application cannot access secret due to RBAC restrictions.
Diagnosis:
```bash
kubectl auth can-i get secrets --as=system:serviceaccount::
kubectl describe rolebinding -n
```
Solution:
```bash
Create appropriate role and binding
kubectl create role secret-reader --verb=get,list --resource=secrets
kubectl create rolebinding app-secret-access \
--role=secret-reader \
--serviceaccount=:
```
Issue 4: Secret Not Updating in Pod
Problem: Updated secret values not reflected in running pods.
Diagnosis:
```bash
Check secret update timestamp
kubectl get secret -o yaml
Check pod creation time
kubectl describe pod
```
Solution:
```bash
Restart deployment to pick up new secret values
kubectl rollout restart deployment/
Or delete pods to force recreation
kubectl delete pod -l app=
```
Issue 5: Secret Size Limitations
Problem: Secret too large error when creating secrets.
Diagnosis:
```bash
Check secret size
kubectl get secret -o yaml | wc -c
```
Solution:
```bash
Split large secrets into multiple smaller secrets
Or use ConfigMaps for non-sensitive large data
Maximum secret size is 1MB
```
Monitoring and Maintenance
Secret Auditing
Enable audit logging for secret operations:
```yaml
audit-policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: Metadata
resources:
- group: ""
resources: ["secrets"]
verbs: ["get", "list", "create", "update", "patch", "delete"]
```
Monitoring Secret Usage
Create monitoring alerts for secret operations:
```bash
Example Prometheus query for secret access
sum(rate(apiserver_audit_total{objectRef_resource="secrets"}[5m])) by (verb, objectRef_name)
```
Regular Secret Rotation
Implement automated secret rotation:
```bash
#!/bin/bash
secret-rotation-cron.sh
SECRETS_TO_ROTATE=("database-credentials" "api-keys" "jwt-secrets")
for secret in "${SECRETS_TO_ROTATE[@]}"; do
echo "Rotating secret: $secret"
# Generate new password
NEW_PASSWORD=$(openssl rand -base64 32)
# Update secret
kubectl patch secret "$secret" -p="{\"stringData\":{\"password\":\"$NEW_PASSWORD\"}}"
# Restart related deployments
kubectl rollout restart deployment -l uses-secret="$secret"
echo "Secret $secret rotated successfully"
done
```
Add to crontab for regular execution:
```bash
Run secret rotation monthly
0 2 1 /path/to/secret-rotation-cron.sh
```
Backup and Recovery
Create secret backup procedures:
```bash
#!/bin/bash
backup-secrets.sh
NAMESPACE=${1:-default}
BACKUP_DIR="/backup/secrets/$(date +%Y%m%d)"
mkdir -p "$BACKUP_DIR"
Backup all secrets in namespace
kubectl get secrets -n "$NAMESPACE" -o yaml > "$BACKUP_DIR/secrets-$NAMESPACE.yaml"
Backup individual secrets
for secret in $(kubectl get secrets -n "$NAMESPACE" -o name); do
secret_name=$(echo "$secret" | cut -d'/' -f2)
kubectl get "$secret" -n "$NAMESPACE" -o yaml > "$BACKUP_DIR/$secret_name.yaml"
done
echo "Secrets backed up to $BACKUP_DIR"
```
Conclusion and Next Steps
Kubernetes secrets are a fundamental component for securing sensitive data in containerized environments. Throughout this comprehensive guide, we've covered everything from basic secret creation to advanced security practices and troubleshooting techniques.
Key Takeaways
1. Security First: Always use secrets for sensitive data instead of hardcoding values
2. Proper Access Control: Implement RBAC to limit secret access to necessary services
3. Regular Rotation: Establish procedures for regular secret rotation
4. Monitoring: Keep track of secret usage and access patterns
5. Backup Strategy: Maintain secure backups of critical secrets
Next Steps
To further enhance your Kubernetes security posture, consider these advanced topics:
1. External Secret Management: Integrate with tools like HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault
2. GitOps with Secrets: Implement secure GitOps workflows using tools like Sealed Secrets or External Secrets Operator
3. Service Mesh Security: Explore mTLS and certificate management with service mesh solutions
4. Compliance: Implement compliance frameworks like SOC 2 or PCI DSS for secret management
5. Advanced Monitoring: Set up comprehensive monitoring and alerting for secret-related security events
Additional Resources
- [Kubernetes Official Documentation on Secrets](https://kubernetes.io/docs/concepts/configuration/secret/)
- [CNCF Security Best Practices](https://www.cncf.io/blog/2019/01/14/9-kubernetes-security-best-practices-everyone-must-follow/)
- [External Secrets Operator](https://external-secrets.io/)
- [Sealed Secrets](https://github.com/bitnami-labs/sealed-secrets)
By following the practices outlined in this guide, you'll be well-equipped to manage Kubernetes secrets securely and effectively in your Linux environment. Remember that security is an ongoing process, and staying updated with the latest best practices and security patches is crucial for maintaining a secure Kubernetes cluster.