Writing a k8s controller in GoLang: delete pods on secret change

The motivation behind writing was to explore how customer controller works in kubernetes. As I was doing it, i felt like doing something that solves a problem. Most of the deployment practically we use has some form of secret mounted with it. It is usually a common practice to iterate those secret every now and then, but one problem that we face is that, after we change the secrets of kubernetes, it does not reflect the pods immediately. It is never a flaw but the idea here is, along with the new deployment of the application, the secrets are going to be changed but for people like me, who wants to see the changes immediately, it can be annoying sometime. One way to solve the problem is to kill the pods associated with the deployment one by one. As the pods are being recreated, it picks the latest secret instead of the old one. Usually developers uses kubectl command to delete the pods but in this blog I am going to write a custom controller using go lang.

Rather than forcing all the pods to recreate, I am going to use an annotation instead, so that I have more control on the pods that I want to restart. I am going to use an annotation ‘sadafnoor.me/pod-delete-on-secret-change’ which is going to contain a list of secret names, for which the pods in the deployment is going to automatically delete and then later recreated by raplicasets.

Install operator-sdk:

brew install operator-sdk
operator-sdk version

Initiate a project:

mkdir label-operator && cd label-operator
operator-sdk init --domain=sadafnoor.me --repo=github.com/sadaf2605/label-operator
operator-sdk create api --group=core --version=v1 --kind=Secret --controller=true --resource=false

Lets write our controller in secret_controller.go file:

package controllers

import (
	"context"
	"strings"

	"github.com/go-logr/logr"
	appv1 "k8s.io/api/apps/v1"
	corev1 "k8s.io/api/core/v1"
	"k8s.io/apimachinery/pkg/runtime"
	ctrl "sigs.k8s.io/controller-runtime"
	"sigs.k8s.io/controller-runtime/pkg/client"

	logf "sigs.k8s.io/controller-runtime/pkg/log"
	"sigs.k8s.io/controller-runtime/pkg/log/zap"
)

type SecretReconciler struct {
	client.Client
	Log    logr.Logger
	Scheme *runtime.Scheme
}

// +kubebuilder:rbac:groups=core,resources=pods,verbs=get;list;watch;update;patch
// +kubebuilder:rbac:groups=core,resources=secrets,verbs=get;list;watch;update;patch

const (
	addPodNameLabelAnnotation = "padok.fr/add-pod-name-label"
	podNameLabel              = "padok.fr/pod-name"
)

var globalLog = logf.Log.WithName("global")

func (r *SecretReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	opts := zap.Options{}
	logger := zap.New(zap.UseFlagOptions(&opts))
	logf.SetLogger(logger)
	// globalLog.Info("Printing at INFO level")
	log := globalLog

	secret := &corev1.Secret{}
	r.Get(ctx, req.NamespacedName, secret)

	dep := &appv1.DeploymentList{}
	r.List(ctx, dep)
	for i := range dep.Items {
		secretChanged := strings.Contains(dep.Items[i].Annotations["sadafnoor.me/pod-delete-on-secret-change"], secret.Name)

		if secretChanged {
			podt := &corev1.Pod{}
			log.Info("Trying to delete all pods that has been using secret.Name: " + secret.Name)
			r.DeleteAllOf(ctx, podt, client.InNamespace(req.NamespacedName.Namespace), client.MatchingLabels(dep.Items[i].Spec.Selector.MatchLabels))
		}

	}

	return ctrl.Result{}, nil
}

func (r *SecretReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&corev1.Secret{}).
		Complete(r)
}

Now lets run this operator controller using

make run

Now we are going to write a deployment to demonstrate that it is working as expected:

Lets write a deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: mariadb
  name: mariadb-deployment
  annotations:
    sadafnoor.me/pod-delete-on-secret-change: "mariadb-root-password"

spec:
  replicas: 1
  selector:
    matchLabels:
      app: mariadb
  template:
    metadata:
      labels:
        app: mariadb
    spec:
      containers:
      - name: mariadb
        image: docker.io/mariadb:10.4
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mariadb-root-password
              key: password
        ports:
        - containerPort: 3306
          protocol: TCP
        volumeMounts:
        - mountPath: /var/lib/mysql
          name: mariadb-volume-1
      volumes:
      - emptyDir: {}
        name: mariadb-volume-1
---
apiVersion: v1
kind: Secret
metadata:
  name: mariadb-root-password
type: Opaque
data:
  password: S3ViZXJuZXRlc1JvY2tzIQ==

Lets apply those changes:

kubectl apply -f deployment_example.yaml

Or we can annotate using:

kubectl annotate deployment.v1.apps/mariadb-deployment sadafnoor.me/pod-delete-on-secret-change=mariadb-root-password=mariadb-root-password

sadaf@CORP-L-0000191 example % kubectl get pods        
NAME                                  READY   STATUS    RESTARTS   AGE
mariadb-deployment-7cb4dff678-h6pdq   1/1     Running   0          18m
my-nginx-6b74b79f57-kpxl6             1/1     Running   0          22m

Let’s the secret and hope that it deletes the pods that has been using it:

apiVersion: v1
kind: Secret
metadata:
  name: mariadb-root-password
type: Opaque
data:
  password: aGVsbG93b3JsZA==

and then

sadaf@CORP-L-0000191 example % kubectl get pods                        
NAME                                  READY   STATUS    RESTARTS   AGE
mariadb-deployment-7cb4dff678-v8fb4   1/1     Running   0          10s
my-nginx-6b74b79f57-gtqpw             1/1     Running   0          117s

That means the pods are being restarted. Wala. I have written my first go code and first kubernetes controller.