Helm for Kubernetes. Handling secrets with SOPS
The first Kubernetes related article I wrote contains a practical introduction to Helm charts. The current one takes a step ahead by showing a possible way to operate with sensitive data inside Helm charts. It also introduces SOPS: Secret OPerationS as a convenient way not to tear your secret out of your repository and keep it side-by-side with the rest of source files while supplying it with privacy. The intention is not to use existing Helm plugins, like helm-secret, but to achieve the same result by yourself and to learn more along the way.
The previous article ends by running an imaginary application having two parts: RabbitMQ server and naive Rest API server talking to RabbitMQ. Both pieces are crafted as Helm charts (where RabbitMQ is professionally maintained and Rest API part is just a bit more than primitive Node.js Express application which feeds RabbitMQ queue from one side and gets messages back when the endpoint is called). Both Helm charts are absolutely neglecting even minimal security requirements since Admin password is sent from outside as a parameter during RabbitMQ installation and stored as plain text in git-repo for the second chart. That is what will be fixed here in a way that secrets continue to be stored in git-repo but in an encrypted way.
SOPS is powerful utility from Mozilla that encrypts configuration files with AES. It has a seamless integration with Cloud providers allowing store encryption key there and to continue using it locally for viewing/editing that files combining it with auto decryption during installation in clouds. While it super useful by itself, the intention of this story is to use GPG key locally to secure configuration file.
Please download GPG package for your OS and install it. Use Kleopatra UI to generate your custom key and register it locally (my certificate has fingerprint 88D3C0D03895A07DB04A393FE6C332F04B58CA6A, yours is different).
Clone the final code from previous article which has two Helm charts (both using unsecure credentials) and start handling secrets properly. First round is RabbitMQ (stored as a copy of professionally maintained chart in rabbitmq subfolder).
Time to secure Admin’s password for RabbitMQ. Let’s put it into separated file:
And encrypt with SOPS command:
- e: encryption operation
- i: in place, i.e. file will be overridden with its encrypted version
- pgp: specifies locally stored encryption key fingerprint (yours will be different). The same fingerprint will be needed during the installation process. Overall, SOPS supports a handy option to specify key fingerprint stored on the cloud provider side
Encrypted version of credentials.yaml is below:
where in #1 the key is left unencrypted contrary to its value. The rest of the file is SOPS signature used in the decryption process.
It is highly advisable to read through SOPS usage documentation section to understand more, especially editing and decryption functionality.
A significant advantage is that it is absolutely safe to store encrypted secrets file in source controls, without fear that values can be stolen by anybody. Only those who have key will be able to get profit from the file. Let’s store the encrypted file in rabbitmq/secrets folder. Sure, there could be many key-value pairs in the file and many such files.
The next task is to improve RabbitMQ Helm chart deployment to reference encrypted secrets. For that the decryption needs to be done:
Above script uses SOPS to decrypt files from secrets folder one-by-one and store it in secrets-decrypted folder. Do not forget to add decrypted folder into .gitignore as only encrypted files should go to source control.
Time to look how Kubernetes defines secrets. Below is manifest sample from documentation page:
#2..#4: secrets itself
#5: corresponding Kubernetes type
#10: secrets name to reference in other Kubernates resources
#11: Kubernetes namespace (plan is to put all resources in non default namespace to eliminate clashes there)
There are one or more decrypted secret files that need to be transformed in Kubernetes secret manifests with Helm charts GO template rendering. Inside rabbitmq/templates folder create my-secrets.yaml:
#1..#3: take the file in secret-decrypted folder
#4..#7: generate Kubernetes secrets part (notice, that secret name is rabbit-secret)
#8: set secrets’ namespace equal to one where whole RabbitMQ getting installed
#11..#13: insert each line in the decrypted file as Kubernetes secrets manifest format expects.
When Helm chart is installed, it setup Kubernetes secrets (named rabbit-secret) with Admin password. Need to link secret to RabbitMQ. For that, reference RabbitMQ Helm chart documentation from Bitnami and find two parameters auth.username and auth.existingPasswordSecret ( it is not possible to use auth.password as it has unsecured password). Thus, a new yaml file (again!) need to be created to set mentioned parameters:
#3: references K8s password as secret named rabbit-secret. In that secret , according to chart documentation, should be key rabbitmq-password holding password itself.
Altogether, it gives superuser/superpassword Admin credentials. As logic is spread between few files, here is git-repo and highlighted files:
Altogether, two lines of bash script is needed to decrypt secrets and install Helm chart:
#1: decrypts secret yaml files stored in rabbitmq folder using key with 88D3C0D03895A07DB04A393FE6C332F04B58CA6A fingerprint
#2: installs Helm chart with name my-rabbitmq in medium namespace using extra file my-values.yaml and ./rabbitmq/ as root folder
Improving the second Helm chart (tiny hand-made “hearbeats” Rest API server) will be simpler since most steps can be borrowed with little modification. Source code is located in the sibling folder serverhb.
Create file credentials.yaml in serverhb/secrets subfolder with the following content:
Absolutely similar SOPS command to inplace file content encryption:
Encrypted version of credentials.yaml:
#1..#2: username and password values are encrypted
This version is safe to commit to git.
Same bash script to decrypt secrets locally will be used here as well. The decrypted version will be put in serverhb/secrets-decrypted subfolder (do not forget to include subfolder to .gitignore).
Remove plain username/password from values.yaml as it is not serious approach for now:
The only Helm chart’s yaml file demands a considerable change now. Before the whole connection string from values.yaml was referenced. With new approach part of it (username and password) is stored inside secrets. The new redaction of this file is below:
#32: old approach to reference the whole connection string (to be removed)
#20..#24: define environment variable USERNAME as reference to k8s secrets named heartbeat-secret and key username inside it
#25..#29: similar approach for environment variable PASSWORD
#31: combine RabbitMQ connection string from newly defined variables and host name (which is equal to other Helm chart name)
And the next step is to send decrypted secrets to Kubernetes by creating secret Kubernetes resources via setting Helm chart template transformation (similar to what was done for RabbitMQ chart above). Inside serverhb/templates folder create my-secrets.yaml:
#1..#3: take the file in secret-decrypted folder
#4..#7: generate Kubernetes secrets part (notice, that secret name is hearbeat-secret, which corresponds to secret referenced in main.yaml)
#8: set secrets’ namespace equal to one where heartbeat service getting installed
#11..#13: insert each line in the decrypted file as Kubernetes secrets manifest format expects (in this case it will be username line and password line).
As logic is spread between a few files, here is git-repo and highlighted files:
And again, two lines of bash script are needed to decrypt secrets and install Helm chart:
Two Helm charts are deployed now:
List of all relevant Kubernetes resources:
Here is the list of all Kubernetes secrets including ours:
To get access to heartbeats server run command:
Get the correct endpoint response in opened browser:
Final Helm charts sources can be found here.
Keeping the whole deployment infrastructure under source control is a crucial requirement. However sensitive data (credentials, hash salt, private keys and many others) is a very special case. SOPS is a great example of an approach that eliminates all security obstacles while keeping secrets along with other code.
