Helm for Kubernetes. Handling secrets with helm-secrets plugin

Yuri Fenyuk
6 min readNov 30, 2022

--

The current chapter is the continuation of Helm exploration journey started with previous ones:

The input for the current chapter is here. The project has two Helm charts (RabbitMQ is maintained by Bitnami and a hand-crafted chart with naive REST API server communicating with RabbitMQ). The previous development step was to introduce add HTTPS certificate to Ingress to make both charts to be accessible via HTTPS. The current step is to reimplement a full custom-made secret handling procedure with one of Helm plugin.

One of GitOps principle encourages using version control systems to store infrastructure declarative configuration. The importance of storing sensitive data securely is extremely important and the convenience of storing it along with other configuration files is extra convenient. In one of my previous chapters, I showed how with help of SOPS and private certificate to encrypt secrets and commit only encrypted secrets to the repository while still having a way to supply decrypted secrets to Helm charts to let them deploy Secrets to Kubernetes. That exercise helped to understand the mechanics but was not too elegant as it demands manual decryption to the temporary file.

Helm charts plugin ecosystem has a dedicated plugin which still use SOPS under the hood to follow the steps I did manually. helm-secrets plugin has great documentation and was deprecated in favor of it successor which needs to be installed:

install Helm plugin

Also, check that SOPS was installed as part of plugin installation or install it separately (better to use the original plugin readme file for details).

My development and demonstration are happening completely on local (Windows! :) ) PC with Kubernetes cluster deployed onto minikube, the should be PGP key installed locally. Please, refer one of my previous chapters for details. Ther plan is to reuse generated key with fingerprint 88D3C0D03895A07DB04A393FE6C332F04B58CA6A from that chapter.

Let’s create minimal .sops.yaml file where local key is referenced (please follow the link to get more information on integrating with different Cloud providers and their key storage parts) and put it at the root of Helm charts, which make it available for all.

key to be used for encryption/decryption

The preparation step is done, so plain secret file encryption can be started. For RabbitMQ chart secret parameter is only one:

unencrypted secret file
encrypt file with secrets

The encrypted result in credentials.yaml:

#1: is the payload of file now with encrypted value (very useful that key is left untouched to allow easy visual identification what is in file)

#8: says that file encrypted with local PGP key

#12: used PGP key fingerprint

Starting from now, there is not need to even keep the plain secret file (credentials.yaml.dec) and better to remove it. When correction to secret (change value, add new pair, or remove old) is needed, the inplace edit command can be used.

Next step is to reference secrets, which Helm plugin decrypts on the fly during installation, using it as input to Kubernetes Secrets. Following plugin documentation and RabbitMQ Helm chart documentation from Bitnami, the resulting secret file for RabbitMQ looks like this:

Setting password for RabbitMQ

#4: name of K8s secret with password (is referenced in my-values.yaml)

#8: Go template function to read secured_rabbitmq_password from chart’s in-memory value storage, where it will be added by the plugin

The install command for this chart is quite long:

helm secrets: helm-secrets plugin adds a wrapper around Helm core binary

install: installing new chart

my-rabbitmq: chart name

-f rabbitmq/my-values.yaml: set of key-value pairs to customize RabbitMQ which were described in detail in the previous chapter (nothing new in the current chapter)

-f rabbitmq/secrets/credentials.yaml: encrypted secrets (with SOPS), which will be read by helm-secrets plugin on the fly and become available as {{ .Values.XXX }} for reference in Helm YAMLs.

Chart installation output

Please notice the very first and last lines tagged with [helm-secrets] informing that the encrypted secret file was found and handled with care.

To test the result, the minikube tunnel command needs to be kept running first. Then https://rabbitmq.localdev.me/ shows Login screen (superuser/superpassword) and the administrative console for RabbitMQ is accessible.

working RabbitMQ

It is time to amend hand crafted Heartbeats server (serverhb) Helm chart.

Unencrypted secrets file
Generate encrypted secret file

Now unencrypted credentials.yaml.dec file should be deleted.

Encrypted file content

The only needed change in Helm chart itself is the structure of my-secrets.yaml, which now looks like following:

#8..9: secret key-value pairs. Value for value is substituted from decrypted on the fly credentials.yaml.

Helm chart install command:

install Helm chart

helm secrets: helm-secrets plugin adds a wrapper around Helm core binary

install: installing new chart

my-serverhb: chart name

-f serverhb/secrets/credentials.yaml: encrypted secrets (with SOPS), which will be read by helm-secrets plugin on the fly and become available as {{ .Values.XXX }} for reference in Helm YAMLs.

Installing output for this Helm chart is shorter, but [helm-secrets] logs are similarly starting and finishing process:

Chart installation output

Running test for Heartbeats server (serverhb) shows that API endpoint returns useful data, meaning that handler code in serverhb Pod has successfully connected to RabbitMQ Pod located in same Kubernetes namespace.

API in action

Let’s overview Kubernetes resources with Lens. Two Pods, one for each Helm chart:

Pods

Quite a lot Secrets:

Secrets

Among them:

heartbeat-secret: Heartbeats server (serverhb) secrets created with the second Helm chart

my-domain-certificate: TLS certificate used in Kubernetes ingress and minikube ingress plugin

my-rabbitmq: RabbitMQ Erlang cookie with name equal to first Helm chart

rabbit-secret: RabbitMQ secret with password

Two Kubernetes ingress entries, making RabbitMQ accessible via https://heartbeats.localdev.me/ and Heartbeats server (serverhb) accessible via https://rabbitmq.localdev.me/. localdev.me is a synonym for localhost (refer to my other article for the trick details)

Finally, Helm charts:

Helm charts

my-certificate: TLS certificate wrapped in a simple Helm chart

my-rabbitmq: minimally customized RabbitMQ Helm chart from Bitnami

my-serverhb: handcrafted Heartbeats server (serverhb)

This is the fifth chapter exploring Helm charts, so let’s sum up current features:

RabbitMQ Helm chart professionally maintained by Bitnami

handcrafted Heartbeats server (serverhb) shows how easy to create own Helm chart

secrets for the above Helm charts are stored in code repository next to charts and encrypted by SOPS (one chapter has bare SOPS utility and current chapter uses Helm plugin)

ingress is setup both Helm charts (see one of the previous chapters)

ingress HTTPS access is done with help of self-signed certificate (see one of the previous chapters). For convenience, this certificate is also wrapped in Helm chart

Using Helm plugin is an easy task and it makes sense to lookup through the plugin list before diving into your own coding journey. Still, it was useful to take a deeper dive into SOPS in one of the previous chapter to understand internal mechanics. Current chapter positive outcome is truly Helm chart approach for the whole functionality.

In addition, it is worth mentioning, despite using locally generated PGP key everywhere in chapters, SOPS has seamless integration with major Cloud providers, so the secret encryption/decryption part can easily be amended to a cloud scenario.

--

--