Kubernetes Operator. Create the one with Kubebuilder.
One of possible way to customize the Kubernetes cluster is to use Operators. They extend Kubernetes capabilities by automating the lifecycle management of applications beyond what’s natively supported. This process is powered by Custom Resource Definitions (CRDs) and Custom Resources (CRs). CRDs allow you to define your own API objects, while CRs are the instances of these definitions.
Operators continuously monitor the state of these CRs and ensure the cluster’s state aligns with the desired specifications. This is achieved through a reconciliation loop, a core concept where the Operator code repetitively checks the current state and works to reconcile it with the desired state defined in the CRs. The ability to automate these tasks makes Operators invaluable for managing complex applications. In this blog, I’ll guide you through the creation of a Kubernetes Operator using Kubebuilder, SDK designed to simplify this process.
If you need a good introduction to Kuberntes Operators, please read following article. My plan is to code working Operator, which has more functionality than the traditional “hello world” Operators. For that purpose, I will reuse my test Golang web-service wrapped into Kubernetes Deployment in previous chapter:
In a nutshell, I tried to find optimal Kuberntes Container memory limit parameters by bombarding my web-service with simultaneous artificial clients requests and changing the limits to avoid OutOfMemory exceptions followed by Container restarting.
Today, I plan to create Kubernetes Operator which periodically runs inside Kubernetes, and during its reconciliation checks whether Pod (I use terms Pod and Container interchangeably here since my only Pod has one Container only) was restarted. If this is the case, the Operator will increase Container’s memory limit a little bit and also increase GOMEMLIMIT runtime variable value (please refer to my earlier chapter with try-and-see approach on how to tune Golang application in terms of memory). I hope the gradual Container parameters increase should heal Pod eventually when it is allocated with enough memory to carry on the workload.
As a humble disclaimer, the decision to tweak Kubernetes Deployment via Kuberntes API is against IaC declarative principle as a minimum, but I find it is Ok for the playground kind of project. In addition, I can again reuse the artifacts from my previous articles and it saves me time.
Within Kubernetes, the following resources will be created:
#1: test Deployment which my Operator will take care of. I will use the one from previous chapter
#2: Custom Resource Definition (CRD) to define properties:
There will be a need to specify which Pod’s label to search for and how big the memory incremental step will be.
#3: Custom Resource (CR) which is the instance of CRD and might have the following values:
#4: Kubernetes Operator wrapped into Kubernetes Deployment. This operator has Goland code with Controller’s Reconcile function, which will be called as new Custom Resource(CR) will be deployed and this CR will be passed as a parameter into the function. The function body should implement the diagram logic presented above.
While these 4 components sound a lot, especially Kuberneter Operator, the powerful scaffolding capability of Kubebuilder comes to the rescue, allowing quick going to the experimentation phase.
Please go to Kubebuider’s github page for installation details and nice documentation to read the basics as these steps are not covered here and the next step is to scaffold a new project:
Those commands created the whole project structure and files with sample Golang code and scripts to do full Operator build and deployment. The output of two commands is in the screenshot:
CRD name is MemoryAdjuster, it has v1 only and is prefixed with group demo1.
As mentioned earlier, the plan is for CRD to have two properties (targetPodLabel and memoryIncrement), so the following modifications are needed(full file path: <root>/api/v1/memoryadjuster_types.go):
To generate CRD as YAML file, run ‘make manifests’command and inspect output:
#3: CustomResourceDefinition kind
#7..#11: different names to use in k8s API (and kubectl) to get/update etc.
#19..#25: two custom properties details
The above step is optional, as Kubebuilder incorporated it in deploy script anyway.
The next step is to create Custom Resource (CR) of instance of CRD MemoryAdjuster. Kubebuilder was so nice to generate a sample here also. In my project it is located on <root>/config/samples/demo_v1_memoryadjuster.yaml and with two new lines looks like:
#9: my Operator going to watch Pod with the label ‘golang-demo’
#10: every time Operator finds fresh Pod termination, it will increase Container request limit on 512Mi
It is time to create CRD and CR in k8s Cluster:
Let’s deploy Operator with an empty Reconcile function to make sure Operator has been called by the Cluster. Kubebuilder has also created an empty implementation:
Modify it to look like the following:
#4: print CR name for which reconciliation is called (there is an assumption here that k8s Namespace has only single CR of MemoryAdjuser kind, which should be Ok for the article)
#7..#10: use k8s API to get details of CR
#8: print CR parameters (it will be used later to conditionally modify Container declaration)
#13: ask k8s to call this reconciliation one minute later in a loop.
The very last step is to use Kubebuilder’s commands (make generate -> make install -> make run) to prepare the Operator and run it (important, to simplify development, the Operator will be run outside of Cluster. When it is ready, use Kubebuilder to wrap it into k8s Deployment and do the actual deployment. After this, the Operator will be truly run inside the Cluster). Terminal output on my laptop looks like this:
Quite a lot of printed text, but there is a clear match between Golang code and what is in the output. Once more, this reconciliation is run each minute as it is asked at the end of the function.
Let’s write more Golang to code the Operator reconciliation logic(please take a second look at the Diagram at the beginning of the article).
Reconcile function going to call new handleOOMPods function call:
that looks like this:
#6..#8: get all Pods that have the label specified in CR
#15..#17: for each such Pod check whether it has been terminated, termination was due to out-of-memory and it happened no longer than 1 minute ago (remember, the Operator runs every minute)
#20: if something is found, increase the resource memory limit in Pod’s declaration
In turn, adjustPodResources function has the following lines:
#5..#8: find Deployment which owns the Pod (Pod belongs to ReplicaSet, which belongs to Deployment)
#10..#13: for each Container in Deployment (I have only one), increase memory with the new adjustContainerMemory function (see below)
#16..#18: update modified Deployment in k8s Cluster (this action results in Pod restarting, but this time it has a bigger memory limit)
How the Pod memory limit is increased is shown in yet another new function:
#3..#5: from CR get memory incremental value
#8..#12: read the current Container’s memory limit, add increment, and update the Container with a new value
#14..#33: for the reliable Golang web-service performance, need to update GOMEMLIMIT value to be a 10% less than the new Container memory limit and also update the Container declaration (refer to one of the previous chapters for details)
True, this was pretty long Golang functions’ calls. Good that visual Digram is in the article to make it easier to grasp the idea. The full Golang code is here and the whole Kubebuilder project is here.
Thus, the Operator plays the role of a watchdog, but which resource it is supposed to take care of? The plan is to borrow k8s test deployment from the previous chapter:
and simulate the test load with k6 performance tool script the same way (50 parallel users, 30 secs session) as in the previous chapter (the script itself).
Here is k8s Deployment for testing. Fully standard and the only thing to notice, is that Pod’s label ‘golang-demo’ should correspond to the specified value in CustomResource (CR), otherwise, the Operator reconciliation filters the Pod out.
#4..#8: the Deployment has only one revision at the moment
#10..#16: the Pod in the Deployment specification initially has 2.8Mb memory limit and GOMEMLIMIT is 2.4Mb
The missing puzzle parts to see the Operator increasing the failed Pod memory limit is to run the Operator with Kubebuilder’s run script command and to bombard it with work simulating script. Let’s do it in parallel.
Below is the development-phase run in Kubebuilder’s session for the Operator, where all Golang code functions flow is executed and the Container’s limits have auto-frown as a result of reconciliation.
The important logs are ‘Container memory limits’ ( with “old”: “2800Mi”, “new”: “3312Mi", which is 512Mi more and equal to CR’s memoryIncrement value), ‘GOMEMLIMIT’ (with “new”: “3125595341”), and ‘Updated resource limits for Deployment’ (with “Deployment”: “golang-demo-deployment”).
Let’s get the test Deployment description now:
#1..#7: now the Deployment has two revisions (since we updated the Container to make it more robust)
#8..#14: new Container memory limit is 3312Mi (which is initial 2800Mi + 512Mi) and GOMEMLIMIT is also grown (the current value is in bytes)
Thus, a practical example of the Operator works and now the Pod has higher limits. Still, it is easy to kill the POd with one more bombardment. Brand new parameters are now even bigger and the Deployment has plus one revision (three in total)
It is possible to kill the Pod a few more times, but after each killing, it gets resurrected with more and more memory limit and, eventually, somewhere at 7..9Gi it is no longer possible.
As a final note, please remember, that the Kubebuilder’s written Operator is still run outside of the Cluster in the terminal session. Please check the documentation on how to deploy it inside the Cluster.
Kubebuilder is a great SDK, that hides the complexity of developing, deploying, and maintaining Kubernetes Operator. Great documentation allows quick focus on ‘todo’ part. The amount of scaffolded code demonstrates how heavy the Operator project might look without Kubebuilder.
