Since etcd holds critical data, protecting it is vital. If an attacker gains access, they can read secrets, modify cluster settings, or even delete everything. Let's explore how we can safeguard etcd using three key security mechanisms: encryption at rest, encryption in transit, and regular backups.
Enabling Data Encryption at Rest
Data in etcd should be encrypted when stored on disk. Without encryption, an attacker with file access can read sensitive information like API tokens, credentials, and configurations. Kubernetes provides a built-in way to ensure that its configuration and data are protected. The first thing to do is to modify the etcd pod specification file, which is typically located at /etc/kubernetes/manifests/etcd.yaml to make sure that encryption is enabled for the data stored in etcd.Steps to Enable Encryption
1. Create an encryption configuration file:
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
providers:
- aescbc:
keys:
- name: key1
secret: c2VjcmV0a2V5dmFsdWU=
- identity: {}Where:
- kind: Specifies the type of Kubernetes object. Here, it's an EncryptionConfiguration
- apiVersion: Specifies the version of the Kubernetes API
- resources: Lists the types of resources that we want to be encrypted. In this example, it’s set to secrets, meaning Kubernetes Secrets will be encrypted
- providers: Defines the encryption providers to use. In this case, aescbc is used, which is a type of encryption algorithm. The keys section contains the encryption keys. The name is a label for the key, and secret is the actual encryption key, base64-encoded
- identity: Acts as a fallback in case the aescbc provider cannot be used.
2. Update the API server configuration:
Next we’ll update the etcd pod specification and add the following flag to the Kubernetes API server:
This will tell the API server to use the encryption configuration file and encrypt the objects specified in the way described in that file. For that, open the /etc/kubernetes/manifests/etcd.yaml file and add the —encryption-provider-config flag to the etcd command. Here’s how to do it:
--encryption-provider-config=/etc/kubernetes/encryption-config.yaml3. Restart the API server:
After applying the encryption config, restart the API server to apply changes.
Encrypting Data in Transit with TLS
Data should also be encrypted while traveling between etcd and other Kubernetes components. Without encryption, attackers can intercept sensitive information. We achieve this using TLS (Transport Layer Security).Steps to Enable TLS
1. Generate TLS certificates: Use OpenSSL or a tool like cfssl to create a certificate authority (CA) and issue server and client certificates.2. Configure etcd to use TLS: Edit the etcd configuration file to include the paths to the TLS certificates
3. Configure Kubernetes components to use TLS: Add these flags to the API server:
--cert-file=/etc/kubernetes/pki/etcd-server.crt \
--key-file=/etc/kubernetes/pki/etcd-server.key \
--client-cert-auth \
--trusted-ca-file=/etc/kubernetes/pki/etcd-ca.crt--etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt \
--etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key \
--etcd-cafile=/etc/kubernetes/pki/etcd-ca.crtRegular Backups to Protect Against Data Loss
Even with strong security, failures happen. A hardware crash or accidental deletion can wipe out etcd data. Regular backups help us recover quickly.Steps to Create and Restore Backups
Use etcd’s built-in snapshot functionality to create backups and restore from them1. Take a snapshot:
ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-snapshot.db \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd-ca.crt \
--cert=/etc/kubernetes/pki/etcd-server.crt \
--key=/etc/kubernetes/pki/etcd-server.key- ETCDCTL_API=3: Specifies the etcdctl API version to use.
- etcdctl snapshot save: The command to take a snapshot of the etcd data.
- /backup/etcd-snapshot.db: The path where the backup file will be saved. Replace this with your desired backup location.
- --endpoints=: Specifies the etcd endpoints to connect to. Replace with the actual addresses of your etcd instances.
- --cacert - The path to the CA certificate used to verify the etcd server certificate.
- --cert - The path to the client certificate used to authenticate the client to the etcd server.
- --key - The path to the client key used to authenticate the client to the etcd server
This creates a full snapshot of the etcd database. ALWAYS, store these backups in a secure and reliable location to ensure you can restore your cluster if needed.
2. Restore from a backup:
ETCDCTL_API=3 etcdctl snapshot restore /backup/etcd-snapshot.db3. Automate backups: Schedule periodic snapshots using a cron job or Kubernetes job.