Learn how to – Forward Kubernetes Logs to Elasticsearch (ELK) using Fluentbit

Learn how to – Forward Kubernetes Logs to Elasticsearch (ELK) using Fluentbit.

Fluent bit is an open source, light-weight log processing and forwarding service. Fluent bit allows to collect logs, events or metrics from different sources and process them. These data can then be delivered to different backends such as Elastic search, Splunk, Kafka, Data dog, InfluxDB or New Relic.

Fluent bit is easy to setup and configure. It gives you full control of what data to collect, parsing the data to provide a structure to the data collected. It allows one to remove unwanted data, filter data and push to an output destination. Therefore, it provides an end to end solution for data collection.

Some wonderful features of fluent bit are:

1. High Performance
2. It is super Lightweight and fast, requires less resource and memory
3. It supports multiple data formats.
4. The configuration file for Fluent Bit is very easy to understand and modify.
5. Fluent Bit has built-in TLS/SSL support. Communication with the output destination is secured.
6. Asynchronous I/O

Fluent Bit is compatible with docker and kubernetes and can therefore be used to aggregate application logs. There are several ways to log in kubernetes. One way is the default stdout logs that are written to a host path”/var/log/containers” on the nodes in a cluster. This method requires a fluent bit DaemonSet to be deployed. A daemon sets deploys a fluent bit container on each node in the cluster.

The second way of logging is the use of a persistent volume. This allows logs to be written and persistent in an internal or external storage such as Cephfs. Fluent bit can be setup as a deployment to read logs from a persistent Volume.

In this Blog, we will look at how to send logs from a Kubernetes Persistent Volume to Elastic search using fluent bit. Once logs are sent to elastic search, we can use kibana to visualize and create dashboards using application logs and metrics.

PREREQUISITES:

First, we need to have a running Kubernetes Cluster. You can use our guides below to setup one if you do not have one yet:

• Install Kubernetes Cluster on Ubuntu with kubeadm
• Install Kubernetes Cluster on CentOS 7 with kubeadm
• Install Production Kubernetes Cluster with Rancher RKE

Secondly, we will need an elastic search cluster setup. You can use elasticsearch installation guide if you don’t have one in place yet. In this tutorial, we will setup a sample elastic search environment using stateful sets deployed in the kubernetes environment. We will also need a kibana instance to help us visualize this logs.

Deploy Elasticsearch

1. Create the manifest file. This deployment assumes that we have a storage class cephfs in our cluster. A persistent volume will be created along side the elastic search stateful set. Modify this configuration as per your needs.

$ vim elasticsearch-ss.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: es-cluster 
spec:
  serviceName: elasticsearch
  replicas: 1
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      containers:
      - name: elasticsearch
        image: docker.elastic.co/elasticsearch/elasticsearch:7.2.0
        resources:
            limits:
              cpu: 1000m
            requests:
              cpu: 100m
        ports:
        - containerPort: 9200
          name: rest
          protocol: TCP
        - containerPort: 9300
          name: inter-node
          protocol: TCP
        volumeMounts:
        - name: data
          mountPath: /usr/share/elasticsearch/data
        env:
          - name: cluster.name
            value: k8s-logs
          - name: node.name
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: discovery.seed_hosts
            value: "es-cluster-0.elasticsearch"
          - name: cluster.initial_master_nodes
            value: "es-cluster-0"
          - name: ES_JAVA_OPTS
            value: "-Xms512m -Xmx512m"
      initContainers:
      - name: fix-permissions
        image: busybox
        command: ["sh", "-c", "chown -R 1000:1000 /usr/share/elasticsearch/data"]
        securityContext:
          privileged: true
        volumeMounts:
        - name: data
          mountPath: /usr/share/elasticsearch/data
      - name: increase-vm-max-map
        image: busybox
        command: ["sysctl", "-w", "vm.max_map_count=262144"]
        securityContext:
          privileged: true
      - name: increase-fd-ulimit
        image: busybox
        command: ["sh", "-c", "ulimit -n 65536"]
        securityContext:
          privileged: true
  volumeClaimTemplates:
  - metadata:
      name: data
      labels:
        app: elasticsearch
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: cephfs
      resources:
        requests:
          storage: 5Gi

Apply this configuration

$ kubectl apply -f elasticsearch-ss.yaml

2. Create an elastic search service

$ vim elasticsearch-svc.yaml
kind: Service
apiVersion: v1
metadata:
  name: elasticsearch 
  labels:
    app: elasticsearch
spec:
  selector:
    app: elasticsearch
  clusterIP: None
  ports:
    - port: 9200
      name: rest
    - port: 9300
      name: inter-node

$ kubectl apply -f elasticsearch.svc

3. Deploy Kibana

$ vim kibana.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kibana
  labels:
    app: kibana
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kibana
  template:
    metadata:
      labels:
        app: kibana
    spec:
      containers:
      - name: kibana
        image: docker.elastic.co/kibana/kibana:7.2.0
        resources:
          limits:
            cpu: 1000m
          requests:
            cpu: 100m
        env:
          - name: ELASTICSEARCH_URL
            value: http://elasticsearch:9200
        ports:
        - containerPort: 5601

---
apiVersion: v1
kind: Service
metadata:
  name: kibana
  labels:
    app: kibana
spec:
  ports:
  - port: 5601
  selector:
    app: kibana

Apply this configuration:

$ kubectl apply -f kibana.yaml

4. We then need to configure and ingress route for the kibana service as follows:

$ vim kibana-ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/tls-acme: "true"
    ingress.kubernetes.io/force-ssl-redirect: "true"
  name: kibana

spec:
  rules:
  - host: kibana.ownyourlife.com.ng
    http:
      paths:
      - backend:
          serviceName: kibana
          servicePort: 5601
        path: /
  tls:
  - hosts:
    - kibana.ownyourlife.com.ng
    secretName: ingress-secret   // This can be created prior if using custom certs

$ kubectl apply -f kibana-ingress.yaml 

Kibana service should now be accessible via https://kibana.ownyourlife.com.ng/

Once we have this setup, We can proceed to deploy fluent Bit.

Step 1: Deploy Service Account, Role and Role Binding

Create a deployment file with the following contents:

$ vim  fluent-bit-role.yaml 
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluent-bit
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluent-bit-read
rules:
- apiGroups: [""]
  resources:
  - namespaces
  - pods
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fluent-bit-read
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fluent-bit-read
subjects:
- kind: ServiceAccount
  name: fluent-bit
  namespace: default

Apply deployment config by running the command below.

kubectl apply -f fluent-bit-role.yaml

Step 2: Deploy a Fluent Bit configMap

This config map allows us to be able to configure our fluent Bit service accordingly. Here, we define the log parsing and routing for Fluent Bit. Change this configuration to match your needs.

$ vim fluentbit-configmap.yaml

apiVersion: v1

kind: ConfigMap

metadata:

  labels:

    k8s-app: fluent-bit

  name: fluent-bit-config
data:

  filter-kubernetes.conf: |

    [FILTER]

        Name                kubernetes

        Match               *

        Kube_URL            https://kubernetes.default.svc:443

        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token

        Kube_Tag_Prefix     kube.var.log

        Merge_Log           On

        Merge_Log_Key       log_processed

        K8S-Logging.Parser  On

        K8S-Logging.Exclude Off
  fluent-bit.conf: |

    [SERVICE]

        Flush         1

        Log_Level     info

        Daemon        off

        Parsers_File  parsers.conf

        HTTP_Server   On

        HTTP_Listen   0.0.0.0

        HTTP_Port     2020
    @INCLUDE input-kubernetes.conf

    @INCLUDE filter-kubernetes.conf

    @INCLUDE output-elasticsearch.conf

  input-kubernetes.conf: |

    [INPUT]

        Name              tail

        Tag               *

        Path              /var/log/*.log

        Parser            json

        DB                /var/log/flb_kube.db

        Mem_Buf_Limit     5MB

        Skip_Long_Lines   On

        Refresh_Interval  10

  output-elasticsearch.conf: |

    [OUTPUT]

        Name            es

        Match           *

        Host            ${FLUENT_ELASTICSEARCH_HOST}

        Port            ${FLUENT_ELASTICSEARCH_PORT}

        Logstash_Format On

        Replace_Dots    On

        Retry_Limit     False

  parsers.conf: |

    [PARSER]

        Name   apache

        Format regex

        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) [(?<time>[^]]*)] "(?<method>S+)(?: +(?<path>[^"]*?)(?: +S*)?)?"" (?<code>[^ ]*) (?<size>[^ ]*)(?: ""(?<referer>[^""]*)"" ""(?<agent>[^""]*)"")?$