Learn how to – How to deploy Redis StatefulSet Cluster in Kubernetes

Learn how to – How to deploy Redis StatefulSet Cluster in Kubernetes.

Running databases in Kubernetes can be a little bit delicate owing to the fact that data needs to be persisted incase the pods die, are restarted or are malfunctioning due to one reason or another. Due to this, most people still choose the virtual machine way to run their databases because among other things, it feels safe and you know exactly where the data is being stored/persisted. But in case you need all of your applications to reside in Kubernetes, you will be delighted to know that it can handle stateful applications such as databases. There are two ways in which you can achieve this. You can either choose to use Deployments or StatefulSets. Now the two have their pros and cons and their differences are showed in the table below:

Due to the advantages that StatefulSets have compared to Deployments such as separate volumes for each of the pods created, let us go ahead and deploy Redis cluster using this kind. It is best practice to use headless services (service without an IP address) when using StatefulSets. This is because we would wish our clients to connect directly to the pods and good thing is that StatefulSets maintain the same DNS identity. So we shall create a headless service to go along with this Redis cluster.

Pre-requisites

You need to have the following to get this accomplished:

• A working Kubernetes Cluster
• kubectl installed locally or somewhere that is connected to your cluster
• Ability to access your cluster
• For this we will also require Dynamic Volume Provisioning in your cluster.

Once you are ready, we shall therefore set off.

Step 1: Create and deploy headless service

In this step, we shall simply create our headless service as follows.

$ vim redis-service.yml
apiVersion: v1
kind: Service
metadata:
  name: redis-service
  namespace: redis
  labels:
    app: redis
spec:
  ports:
    - port: 6379
  clusterIP: None
  selector:
    app: redis

As you can notice, the service has its clusterIP set to “None” and hence will prevent the service from having an IP giving clients the ability to connect directly to what lies behind it.

With the clean service created, proceed to deploy it as shown below. We will create the “redis” namespace then apply the file. Remember that you can choose any namespace of your choice here.

$ kubectl create ns redis
$ kubectl apply -f redis-service.yml
service/redis-ss created

Step 2: Create and deploy the configuration via a ConfigMap

We shall use a ConfigMap to add configuration parameters dynamically to our Redis Master and Replicas. Create a ConfigMap as follows:

$ vim redis-configmap.yml
apiVersion: v1
kind: ConfigMap
metadata:
  name: redis-ss-configuration
  namespace: redis
  labels:
    app: redis
data:
  master.conf: |
    maxmemory 400mb
    maxmemory-policy allkeys-lru
    maxclients 20000
    timeout 300
    appendonly no
    dbfilename dump.rdb
    dir /data
  slave.conf: |
    slaveof redis-ss-0.redis-ss.redis 6379
    maxmemory 400mb
    maxmemory-policy allkeys-lru
    maxclients 20000
    timeout 300
    dir /data

The data section of the ConfigMap has the two configurations for master and replica which we shall direct them to the right pods depending on the StatefulSets’ ordinal numbers. For your information, StatefulSets assign a sticky identity called an ordinal number starting from zero to each Pod instead of assigning random IDs for each replica Pod. This is as was clarified on the table in the introductory section. You can add more of your configurations on this part as needed. Once you are ready/comfortable with the configs, let’s go ahead and deploy it right away.

$ kubectl apply -f redis-configmap.yml
configmap/redis-configuration-ss created

Step 3: Create and deploy the StatefulSet

We are now in the interesting part of this meal. All of the other parts are ready and we will just plug in the engine and we will be ready to hit the road. Create a new file and fill it with the following StatefulSet configuration then we explain what it does.

vim redis-statefulset.yml

Paste and modify the contents below where applicable.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis-ss
  namespace: redis
spec:
  serviceName: "redis-service"
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      initContainers:
      - name: init-redis
        image: redis:latest
        command:
        - bash
        - "-c"
        - |
          set -ex
          # Generate redis server-id from pod ordinal index.
          [[ `hostname` =~ -([0-9]+)$ ]] || exit 1
          ordinal=${BASH_REMATCH[1]}
          # Copy appropriate redis config files from config-map to respective directories.
          if [[ $ordinal -eq 0 ]]; then
            cp /mnt/master.conf /etc/redis-config.conf
          else
            cp /mnt/slave.conf /etc/redis-config.conf
          fi
        volumeMounts:
        - name: redis-claim
          mountPath: /etc
        - name: config-map
          mountPath: /mnt/
      containers:
      - name: redis
        image: redis:latest
        ports:
        - containerPort: 6379
          name: redis-ss
        command:
          - redis-server
          - "/etc/redis-config.conf"
        volumeMounts:
        - name: redis-data
          mountPath: /data
        - name: redis-claim
          mountPath: /etc
      volumes:
      - name: config-map
        configMap:
          name: redis-ss-configuration                  
  volumeClaimTemplates:
  - metadata:
      name: redis-claim
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1Gi
  - metadata:
      name: redis-data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1Gi

When using StorageClass for PV creation update volumeClaimTemplates section like below.

volumeClaimTemplates:
  - metadata:
      name: redis-claim
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1Gi
      storageClassName: rook-cephfs
  - metadata:
      name: redis-data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1Gi
      storageClassName: rook-cephfs

You can list configured Storage Classes in your cluster with the command below:

$ kubectl get sc
NAME              PROVISIONER                     RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
rook-ceph-block   rook-ceph.rbd.csi.ceph.com      Delete          Immediate           true                   204d
rook-cephfs       rook-ceph.cephfs.csi.ceph.com   Delete          Immediate           true                   204d

In our Statefulset configuration, the Ordinal Number will be 0 for the first pod here and it will be named “redis-ss-0” because the name of the StatefulSet is “redis-ss”

All the way from top, we have declared that we are creating a StatefulSet and we are picking the headless service called “redis-service” that we created in Step 1 as the main entry point. We will be using an initContainer to plug the configuration files to the pods. Init containers are run before the app containers are started and once it completes its task, it goes off and hands over the mantle to the main pod. In this case, the init container is instructed to check the ordinal number of the pod. Remember the “ordinal number” we explained before, here is one of the instances it becomes useful. So we are saying that, in case the ordinal number is zero i.e “redis-ss-0” (notice the tailing zero), pick the “master.conf” configuration from the ConfigMap (now available at /mnt in the initcontainer) and copy it to “/etc/redis-config.conf” file in the main pod. This configuration will have the redis master configs. In case it has any other ordinal number like 1 from the tailing number like “redis-ss-1” then pick the “slave.conf” (now also available at /mnt in the initcontainer) file from the ConfigMap and copy it to “/etc/redis-config.conf” of that new Pod. Pretty Cool!!

You may now be wondering and asking how the ConfigMap config is found at “/mnt” path. As you can see from the VolumeMounts section, we have the name “config-map” representing a mount point at the “/mnt” path. When we check the “volumes” section of the file as well, we see that there is a volume also known as “config-map” that is attached to the ConfigMap called “redis-ss-configuration” we created in Step 2. Basically, the contents of the ConfigMap (master.conf and slave.conf) will be dropped in the volume called “config-map” which will be later mounted at the path “/mnt” and its contents accessed. That is how we get the data/files (master.conf and slave.conf) into the initContainer. Another thing you will notice is that the initContainer and the main pod share one of the volumes and mounted on the same path. This is the “redis-claim” volume. This is correct so that the files transferred from the initContainer can be found by the main pod when it starts and mounts the volume into its directory hierarchy which it “/etc” in this case. Depending on the ordinal number of the pod, the initContainer will copy either master.conf or slave.conf into “/etc/redis-config.conf”.

Once that is done, there is a redis command that will start redis using the “/etc/redis-config.conf” file copied from the initContainer. The command here is:

redis-server /etc/redis-config.conf

Later in the file, we see the “volumeClaimTemplates” which is a way the StatefulSet will create persistent volumes dynamically. This is why one of the requirements for this deployment is the ability of your cluster to create persistent volumes dynamically. In case this is not possible, you will have to create persistent volumes with the same names as the ones in “volumeClaimTemplates” then apply this StatefulSet. The persistent volumes here will be used by the main pod to store its data at “/data” as well as the configurations at “/etc”. We hope the explanation was clear enough. Finally, apply the file like we have done with the others before:

$ kubectl apply -f redis-statefulset.yml
statefulset.apps/redis-ss created

Step 4: Checking if all we have done is working

By now, should be good to go. And the only way to know is to simply check if everything was successfully deployed:

Check the ConfigMap

$ kubectl get configmap -n redis
NAME                     DATA   AGE
redis-ss-configuration   2      3h35m

Check the Service

$ kubectl get svc -n redis
NAME       TYPE        CLUSTER-IP     EXTERNAL-IP      PORT(S)         AGE
redis-service   ClusterIP   None           <none>       6379/TCP     3h40m

Check the StatefulSet

$ kubectl get statefulset -n redis
NAME       READY   AGE
redis-ss   1/1     3h43m

Check the Pods created from the StatefulSet

$ kubectl get pods -n redis
NAME                     READY   STATUS    RESTARTS   AGE
redis-ss-0               1/1     Running   0          3h42m

Check the Persistent Volumes

$ kubectl get pvc -n redis
redis-claim-redis-ss-0   Bound    pvc-8a30a867-8b3e-49c2-ac1d-be05d8fe3255   1Gi        RWO            standard       3h40m
redis-data-redis-ss-0    Bound    pvc-04cd967c-93ec-457e-9cef-582aabae496a   1Gi        RWO            standard       3h40m

Let us now exec into the pod and check that the configuration we want are the ones we get.

$ kubectl exec -it redis-ss-0 -c redis -n redis -- /bin/bash
bash-5.1#

Once inside, enter into redis cli interface:

bash-5.1# redis-cli

Find out if the timeout matches the one on the config file

127.0.0.1:6379> config get timeout
1) "timeout"
2) "300"

Find out if the maxclients matches the one on the config file

127.0.0.1:6379> config get maxclients
1) "maxclients"
2) "20000"

Cool! It seems like we are out of the woods .

Step 5: See if replication works

Now that the master is functioning well, let us increase the replicas and see if the master replicates to the replica pod. Navigate to the StatefulSet file we created and edit the replicas to 2 from 1 as follows then apply the file.

$ vim redis-statefulset.yml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis-ss
  namespace: redis
spec:
  serviceName: "redis-service”
  replicas: 2 ##<<<<< Change to 2
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      initContainers:
      - name: init-redis
        image: redis:latest
        command:
        - bash
        - "-c"
        - |
          set -ex
          # Generate redis server-id from pod ordinal index.
          [[ `hostname` =~ -([0-9]+)$ ]] || exit 1
          ordinal=${BASH_REMATCH[1]}
          # Copy appropriate redis config files from config-map to respective directories.
          if [[ $ordinal -eq 0 ]]; then
            cp /mnt/master.conf /etc/redis-config.conf
          else
            cp /mnt/slave.conf /etc/redis-config.conf
          fi
        volumeMounts:
        - name: redis-claim
          mountPath: /etc
        - name: config-map
          mountPath: /mnt/
      containers:
      - name: redis
        image: redis:latest
        ports:
        - containerPort: 6379
          name: redis-ss
        command:
          - redis-server
          - "/etc/master.conf"
        volumeMounts:
        - name: redis-data
          mountPath: /data
        - name: redis-claim
          mountPath: /etc
      volumes:
      - name: config-map
        configMap:
          name: redis-ss-configuration                  
  volumeClaimTemplates:
  - metadata:
      name: redis-claim
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1Gi
  - metadata:
      name: redis-data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1Gi

Apply the file

$ kubectl apply -f redis-statefulset.yml
statefulset.apps/redis-ss configured

Check the pods to verify if a new one has been added

$ kubectl get pods -n redis
NAME                     READY   STATUS    RESTARTS   AGE
redis-ss-0               1/1     Running   0          3h42m
redis-ss-1               1/1     Running   0          131m

Check the persistent volumes to verify if a new ones (for both templates) have been added for the new pod

$ kubectl get pvc -n redis
redis-claim-redis-ss-0   Bound    pvc-8a30a867-8b3e-49c2-ac1d-be05d8fe3255   1Gi        RWO            standard       3h40m
redis-claim-redis-ss-1   Bound    pvc-6574b160-03a8-48f5-a989-d780d17363fa   1Gi        RWO            standard       3h51m
redis-data-redis-ss-0    Bound    pvc-04cd967c-93ec-457e-9cef-582aabae496a   1Gi        RWO            standard       3h40m
redis-data-redis-ss-1    Bound    pvc-6eadf86f-36b4-4985-9b78-1c7faa402cbb   1Gi        RWO            standard       3h51m

Oh yeah!! There they are. Let us now log into the replica and check replication status information

kubectl exec -it redis-ss-1 -c redis -n redis -- /bin/bash

Once inside, run the info command to see if the replica is connected to the master successfully.

bash-5.1# redis-cli
127.0.0.1:6379> info replication
# Replication
role:slave
master_host:redis-ss-0.redis-ss.redis
master_port:6379
master_link_status:up
master_last_io_seconds_ago:5
master_sync_in_progress:0
slave_read_repl_offset:18970
slave_repl_offset:18970
slave_priority:100
slave_read_only:1
replica_announced:1
connected_slaves:0
master_failover_state:no-failover
master_replid:2cfc7fdcdf3322c981250f854235ff0b92b39dfa
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:18970
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:7519
repl_backlog_histlen:11452
127.0.0.1:6379>

And there we have it. One last thing to note is that clients can connect to the redis instances using their DNS Names. For example, to reach the instances deployed in this guide, the DNS names would be:

• Master DNS Name: redis-ss-0.redis-service.redis or redis-service-1.redis-ss.redis.svc.cluster.local

The format is:

[pod-name].[service-name].[namespace].svc.cluster.local or [pod-name].[service-name].[namespace]

Salutations

It has been fun so far and we hope you did enjoy as much as we did. We would like to continue thanking you for the tremendous support, your readership and the positive comments we continue to receive. That said, let me wish you a wonderful time and a period of success, luck, health and happiness.

Reference:

• Kubernetes StatefulSets Documentation

Other guides of similar content include:

• How To Install Redis on Debian
• Install and Configure Redis Server on RHEL 8 / CentOS 8
• How To Run Redis in Podman / Docker Container
• Monitor Redis Server with Prometheus and Grafana in 5 minutes