Helm for Kubernetes. GitOps with Argo CD.
The current chapter is the continuation of Helm exploration journey started with previous ones:
The input for the current chapter is here. The project has three Helm charts (RabbitMQ is maintained by Bitnami, a handcrafted chart with naive REST API server communicating with RabbitMQ and a utility Helm chart with TLS certificate used for HTTPS access via Ingress). To simplify maintenance charts are grouped with helmfile and divided into two environments (dev and prod) each of it has distinctive parameters.
Ok, so what is GitOps? This is something about Operations or supporting the Production environment in other words. And Git here plays a central role. Operations staff were jealous of Developers who are able to cope with massive projects with the help of Git and decided to use similar approach as well.
4 main principles of GitOps:
- Declarative Configuration — nothing to say more as everything in Kubernetes ecosystem is declarative
- Version Controlled, Immutable Storage — Git is a nice tool, and utilizing it for Operations is always a great idea
- Automatic Pull Operations — an important one, if considering traditional operations schema when one of the continuous deployment server (CD) orchestrates Production environment (PE). It is Ok until it has steady access to both Repository and PE. Still, CD is a spare one, as its functions can be delegated to some program injected in PE itself. Repository (Git Repository in this case) knows what to deploy and PE knows where to deploy and can query Repository about details.
- Continuous Reconciliation — another important runtime phase principle that obligates injected into PE program to monitor Repository and reflect changes in PE.
Argo CD is a prominent example of that injected into PE program which pulls Git Repository for the declarative description of what needs to be deployed in the Kubernetes cluster from one side and monitors what is currently deployed in the cluster from another side. If something new appears in Git Repository, Argo CD will deploy that new in the cluster. It is important to visualize in mind that there's no need to push something toward k8s cluster anymore, just amend the declaration in Git Repository and Argo CD will shortly pull changes into k8s cluster.
This chapter's plan is to move all three Helm chart's descriptions to Argo CD (well, not exactly, as Argo CD has its own k8s resource definition called Application) and let it take care of deploying it. In some sense, helmfile and Argo CD AppProject are doing the same by grouping of individual parts. But nobody can stop me to continue using helmfile to deploy Argo CD itself. And, by adding AppProject as the second Helm chart, an updated helmfile is:
#1 .. #3: reference public Argo CD Helm chart repository.
#9 .. #15: first Helm chart in release is Argo CD, which uses the remote repository referenced above and overrides default values with customer one specified in #15. So, this chart is the core part of GitOps paradigm implementation and runs a set of k8s controllers, CDR, etc. directly in k8s cluster.
#17.. #22: second Helm chart is a new hand-crafted Argo CD specific chart, which describes all business-related charts and lets Argo CD take care of the whole my solution I have been developing through 5 previous chapters (Rabbit MQ, Rest API tine server and TLS certificate). The chart itself is located in the sibling folder (#18), going to be deployed in prod-medium namespace (#19) and its content is described in the file argocd-deployment.yaml (#22)
Let’s review what needs to be done to customize Argo CD installation. All preferences are in values/argocd.yaml.
Luckily Argo CD supports Helm charts out of the box. In previous chapter I have started to use Helm chart secrets plugin for decrypting secrets with SOPS on the fly. Up until now, that had happens on local machine, but today decryption needs to be done by Argo CD directly in Kubernetes cluster. Thus, this plugin should be deployed as part of Argo CD deployment. Page ArgoCD Integration gives very detailed instructions.
In short (believe me, it took me many try-and-fail attempts) this is done with the help of k8s Pod initialization containers, which download secrets plugin, SOPS, kubectl and even curl, copy binaries to custom-tools folder and mount that folder to Argo CD Repo container. In addition age encryption key, preloaded into k8s as secrets also referenced in Argo CD Repo container. A careful reading of integration documentation is a key to success.
Here is that long Argo CD customization file:
#13 .. #22: supporting on Argo CD server Helm secret plugin file references schemas.
#50 .. #78: download plugin's mandatory binaries and store them in /custom-tools folder.
#46 .. #48: mount that folder to repo server.
#80 .. #90: create k8s ServiceAccount which has access to all k8s secrets in cluster. It is important since the plugin uses age secret to decrypt secrets files during installation.
Time to run that deployment (let’s temporarily comment argocd-deployment-production Helm chart in helmfile) and get Argo CD installed in k8s cluster:
Deployment output instructions how to get admin password and after exposing Argo CD server UI (namespace argo, service argocd-server), the login to UI shows empty Argo CD:
Moving to the second Helm chart in helmfile:
argocd-deployment (can be suffixed with environment name, like production, development etc.) is used to create one Argo CD AppProject for a particular environment (here is production) and set of Argo CD Applications. Same as in previous chapters, the plan is to install business value components wrapped in Helm charts: Rabbit-MQ, heartbeats server and tlscert which were copied from the previous chapter without any modification.
And, to let argocd-deployment-production Helm chart know it, the values/argocd-deployment.yaml catalogue them:
So, this chart lists all 3 business charts. To understand, what is happening inside argocd-deployment, let’s look inside its structure and some file content:
appproject.yaml file creates one Argo CD AppProject instance:
#13 .. #15: deploys AppProject into this k8s cluster and namespace specified in values.yaml.
application.yaml is way more interesting:
#1: it iterates over applications that need to be included in the project, in our case 3 business charts.
#4: it creates Argo CD Application resource (for each business chart here and below).
#6: it builds name of application by concatenating destination namespace and application.name. Given this value.yaml, the result names will be prod-medium-tls-certificate, prod-medium-rabbit-mq, prod-medium-server-heartbeats.
#13: each application will be part of AppProject defined by appproject.yaml.
#20: each application will be deployed in specified in values.yaml k8s namespace.
#21 .. #22: git repo URL (including path inside repo) to Helm chart. It is an important feature of Argo CD, it goes to git repo itself and get k8s resources declaration (in this case, it is Helm chart, but due to the fact Argo CD supports Helm chart out of the box and we have added support of Helm secrets plugin, everything should work)
#25 .. : Argo CD Helm specific section
#28 .. #30: for each business chart, add one or more value files. Given this value.yaml, result application will have tls-certificate.yaml, rabbit-mq.yaml, server-heartbeats.yaml for each corresponding chart.
#31..#34: same but for secrets, which are encrypted with age-key.
#33: it is a workaround for Helm secret plugin used inside Argo CD. Please refer https://github.com/jkroepke/helm-secrets/wiki/ArgoCD-Integration#external-chart-and-local-values. Stressing here, that this line decomposition is very important!
Let’s look inside one of secrets YAML:
In comparison to the previous chapters, I needed to move from using PGP keys to age key (#20 is empty now. but #8 is full), because first paragraph of Helm secrets plugin. Still, SOPS commands to create encrypted secret files are similar.
Understood all internal mechanics and what is what in both Helm charts, which form helmfile, let’s run helmfile sync once more. Now both charts are installed:
Fingers crossed, but quickly UI gets updated with all 3 business applications parts:
Detailed application resource diagram in Argo CD is so exciting:
And, the final test for production environment is to go to servers with the browser (need to run ‘minikube tunnel’ first to route all ingresses):
Argo CD has very detailed documentation of its capabilities. GitOps is in full power! Among of features that impressed me is self-healing (try to kill some parts of application (not only Pods, try Service or Secret for instance) and see how it is recreated) and reconciliation if the git repo changes.
Zero changes in 3 business charts are very important and show a relatively simple transition from the previous chapter (where everything were covered with single helmfile and gets deployed from the host PC). in this chapter, helmfile is only left to deploy Argo CD itself and service Helm chart to generate proper Argo CD Application letting Argo CD take care of 3 business charts.
I continue to support a different environment called development. That will completely reuse charts folder content, but envs folder has a new branch for development:
The helmfile here is simpler and only has argocd-deployment-development chart, because only one Argo CD instance is enough for the whole cluster, and better to keep it in production branch (that also results in missing argocd.yaml customization file in development folder):
#9: this chart has development suffix.
All values files for 3 business charts are almost the same (to follow best practices, they should have different param values) except ingress values which are https://dev-rabbitmq.localdev.me and https://dev-heartbeats.localdev.me.
Let’s deploy with helmfile sync and check that Argo CD has twice more applications under its control:
All sources can be found on github.
GitOps approach is rather difficult to overestimate for DevOps staff. Argo CD is equally popular as its competitor Flux, according to CNCF. Both initiatives have many useful features included out of the box and the current chapter is only scratched it a bit. Still, Argo CD fits very well in my rather naive project and levels it up.
