Helm for Kubernetes. Handling secrets with helm-secrets plugin
The current chapter is the continuation of Helm exploration journey started with previous ones:
The input for the current chapter is here. The project has two Helm charts (RabbitMQ is maintained by Bitnami and a hand-crafted chart with naive REST API server communicating with RabbitMQ). The previous development step was to introduce add HTTPS certificate to Ingress to make both charts to be accessible via HTTPS. The current step is to reimplement a full custom-made secret handling procedure with one of Helm plugin.
One of GitOps principle encourages using version control systems to store infrastructure declarative configuration. The importance of storing sensitive data securely is extremely important and the convenience of storing it along with other configuration files is extra convenient. In one of my previous chapters, I showed how with help of SOPS and private certificate to encrypt secrets and commit only encrypted secrets to the repository while still having a way to supply decrypted secrets to Helm charts to let them deploy Secrets to Kubernetes. That exercise helped to understand the mechanics but was not too elegant as it demands manual decryption to the temporary file.
Helm charts plugin ecosystem has a dedicated plugin which still use SOPS under the hood to follow the steps I did manually. helm-secrets plugin has great documentation and was deprecated in favor of it successor which needs to be installed:
Also, check that SOPS was installed as part of plugin installation or install it separately (better to use the original plugin readme file for details).
My development and demonstration are happening completely on local (Windows! :) ) PC with Kubernetes cluster deployed onto minikube, the should be PGP key installed locally. Please, refer one of my previous chapters for details. Ther plan is to reuse generated key with fingerprint 88D3C0D03895A07DB04A393FE6C332F04B58CA6A from that chapter.
Let’s create minimal .sops.yaml file where local key is referenced (please follow the link to get more information on integrating with different Cloud providers and their key storage parts) and put it at the root of Helm charts, which make it available for all.
The preparation step is done, so plain secret file encryption can be started. For RabbitMQ chart secret parameter is only one:
The encrypted result in credentials.yaml:
#1: is the payload of file now with encrypted value (very useful that key is left untouched to allow easy visual identification what is in file)
#8: says that file encrypted with local PGP key
#12: used PGP key fingerprint
Starting from now, there is not need to even keep the plain secret file (credentials.yaml.dec) and better to remove it. When correction to secret (change value, add new pair, or remove old) is needed, the inplace edit command can be used.
Next step is to reference secrets, which Helm plugin decrypts on the fly during installation, using it as input to Kubernetes Secrets. Following plugin documentation and RabbitMQ Helm chart documentation from Bitnami, the resulting secret file for RabbitMQ looks like this:
#4: name of K8s secret with password (is referenced in my-values.yaml)
#8: Go template function to read secured_rabbitmq_password from chart’s in-memory value storage, where it will be added by the plugin
The install command for this chart is quite long:
helm secrets: helm-secrets plugin adds a wrapper around Helm core binary
install: installing new chart
my-rabbitmq: chart name
-f rabbitmq/my-values.yaml: set of key-value pairs to customize RabbitMQ which were described in detail in the previous chapter (nothing new in the current chapter)
-f rabbitmq/secrets/credentials.yaml: encrypted secrets (with SOPS), which will be read by helm-secrets plugin on the fly and become available as {{ .Values.XXX }} for reference in Helm YAMLs.
Please notice the very first and last lines tagged with [helm-secrets] informing that the encrypted secret file was found and handled with care.
To test the result, the minikube tunnel command needs to be kept running first. Then https://rabbitmq.localdev.me/ shows Login screen (superuser/superpassword) and the administrative console for RabbitMQ is accessible.
It is time to amend hand crafted Heartbeats server (serverhb) Helm chart.
Now unencrypted credentials.yaml.dec file should be deleted.
The only needed change in Helm chart itself is the structure of my-secrets.yaml, which now looks like following:
#8..9: secret key-value pairs. Value for value is substituted from decrypted on the fly credentials.yaml.
Helm chart install command:
helm secrets: helm-secrets plugin adds a wrapper around Helm core binary
install: installing new chart
my-serverhb: chart name
-f serverhb/secrets/credentials.yaml: encrypted secrets (with SOPS), which will be read by helm-secrets plugin on the fly and become available as {{ .Values.XXX }} for reference in Helm YAMLs.
Installing output for this Helm chart is shorter, but [helm-secrets] logs are similarly starting and finishing process:
Running test for Heartbeats server (serverhb) shows that API endpoint returns useful data, meaning that handler code in serverhb Pod has successfully connected to RabbitMQ Pod located in same Kubernetes namespace.
Let’s overview Kubernetes resources with Lens. Two Pods, one for each Helm chart:
Quite a lot Secrets:
Among them:
heartbeat-secret: Heartbeats server (serverhb) secrets created with the second Helm chart
my-domain-certificate: TLS certificate used in Kubernetes ingress and minikube ingress plugin
my-rabbitmq: RabbitMQ Erlang cookie with name equal to first Helm chart
rabbit-secret: RabbitMQ secret with password
Two Kubernetes ingress entries, making RabbitMQ accessible via https://heartbeats.localdev.me/ and Heartbeats server (serverhb) accessible via https://rabbitmq.localdev.me/. localdev.me is a synonym for localhost (refer to my other article for the trick details)
Finally, Helm charts:
my-certificate: TLS certificate wrapped in a simple Helm chart
my-rabbitmq: minimally customized RabbitMQ Helm chart from Bitnami
my-serverhb: handcrafted Heartbeats server (serverhb)
This is the fifth chapter exploring Helm charts, so let’s sum up current features:
RabbitMQ Helm chart professionally maintained by Bitnami
handcrafted Heartbeats server (serverhb) shows how easy to create own Helm chart
secrets for the above Helm charts are stored in code repository next to charts and encrypted by SOPS (one chapter has bare SOPS utility and current chapter uses Helm plugin)
ingress is setup both Helm charts (see one of the previous chapters)
ingress HTTPS access is done with help of self-signed certificate (see one of the previous chapters). For convenience, this certificate is also wrapped in Helm chart
Using Helm plugin is an easy task and it makes sense to lookup through the plugin list before diving into your own coding journey. Still, it was useful to take a deeper dive into SOPS in one of the previous chapter to understand internal mechanics. Current chapter positive outcome is truly Helm chart approach for the whole functionality.
In addition, it is worth mentioning, despite using locally generated PGP key everywhere in chapters, SOPS has seamless integration with major Cloud providers, so the secret encryption/decryption part can easily be amended to a cloud scenario.
