Learn how to – Install Elastic Stack 8 (ELK 8) on RHEL 8|CentOS 8

Learn how to – Install Elastic Stack 8 (ELK 8) on RHEL 8|CentOS 8.

ELK stack consists of 3 main open-source components; Elasticsearch, Logstash, and Kibana which work together to allow users to collect, analyze and visualize logs. The role of each component in the ELK stack is explained below:

• Elasticsearch – This is the core of the Elastic software. Elasticsearch is a search and analytics engine. In the ELK stack, it is used to store incoming logs from Logstash and offer the ability to search the logs in real time
• Logstash – This package collects data, transforms logs incoming from multiple sources simultaneously and sends it to storage.
• Kibana – Kibana is a graphical tool for visualizing data. It is used to generate charts and graphs to make sense of the raw data in your database.

The ELK stack can be used together with Beats for log collection. Beats can be defined as lightweight data shippers which allow multiple data sources/indices and send them to Logstash or Elasticsearch. There are several beat packages that include;

• Metricbeat – It collects metrics from systems and services including CPU, memory usage, and load, as well as other data statistics from network data and process data, before being shipped to either Logstash or Elasticsearch directly.
• Auditbeat – It is used to collect Linux audit framework data and monitor file integrity, before being shipped to either Logstash or Elasticsearch directly.
• Filebeat – Its purpose is to forward and files, usually in either .log centralise logs and or .json format.
• Heartbeat – It is used for active probing to determine whether services are available.
• Packetbeat – It supports a collection of network protocols from the application and lower-level protocols, databases and key-value stores, including HTTP, DNS, Flows, DHCPv4, MySQL and TLS. It helps identify suspicious network activities.
• Winlogbeat – It is intended to manage Windows event logs.

This guide offers an in-depth illustration of how to install and use the Elastic Stack 8 (ELK 8) on RHEL 8 / CentOS Stream 8.

1. Install Java on RHEL 8 / CentOS Stream 8

Elasticsearch requires Java 8, 11 installed on the system. If you already have Java installed, you can skip this step. Otherwise, install Java 8 using the command:

sudo yum -y install java-11-openjdk java-11-openjdk-devel

Once installed, verify the Java version.

$ java -version
openjdk version "11.0.15" 2022-04-19 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.15+9-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.15+9-LTS, mixed mode, sharing)

2. Add the Elastic Stack 8.x Repositories on RHEL 8 / CentOS Stream 8

The ELK stack 8 and Beats can be downloaded from the official Elastic Stack 8.x repositories which can be added to the system using the commands below:

First, import the GPG key signing for the repository.

sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

Now add the Elastic Stack 8.x repository on RHEL 8 / CentOS Stream 8.

cat <<EOF | sudo tee /etc/yum.repos.d/elasticsearch.repo
[elasticsearch-8.x]
name=Elasticsearch repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF

Once added, update the YUM package index.

sudo yum clean all
sudo yum makecache

3. Install and Configure Elasticsearch 8 on RHEL 8 / CentOS 8

We will install the latest version of Elasticsearch from the added repositories using the command:

sudo yum install elasticsearch

Dependency Tree:

Last metadata expiration check: 0:00:07 ago on Mon 21 Mar 2022 05:27:10 AM EDT.
Dependencies resolved.
================================================================================
 Package             Architecture Version         Repository               Size
================================================================================
Installing:
 elasticsearch       x86_64       8.1.0-1         elasticsearch-8.x       492 M

Transaction Summary
================================================================================
Install  1 Package

Total download size: 492 M
Installed size: 1.0 G
Is this ok [y/N]: y

Configure Elasticsearch by editing the below YAML file.

sudo vi /etc/elasticsearch/elasticsearch.yml

In the file, make the below adjustments

# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#

# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 0.0.0.0
. . .

# --------------------------------------------------------------------------------

# Enable security features
xpack.security.enabled: false

xpack.security.enrollment.enabled: false

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: false
  keystore.path: certs/http.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: false

If you set a custom data path for Elasticsearch, you need to disable SELinux or set it in permissive mode for the path to be accessible.

sudo setenforce 0

Configure the JVM options by setting the memory limits:

$ sudo vi /etc/elasticsearch/jvm.options
-Xms1g
-Xmx1g

If the system has low memory, configure it to use the least memory:

-Xms256m
-Xmx512m

Start and enable Elasticsearch 8 using the command:

sudo systemctl start elasticsearch
sudo systemctl enable elasticsearch

Verify if Elasticsearch is responding to queries:

$ curl -X GET localhost:9200
{
  "name" : "localhost.localdomain",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "_yNoaSr4SeShhkRNDkDC6A",
  "version" : {
    "number" : "8.1.0",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "3700f7679f7d95e36da0b43762189bab189bc53a",
    "build_date" : "2022-03-03T14:20:00.690422633Z",
    "build_snapshot" : false,
    "lucene_version" : "9.0.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

4. Install and Configure Logstash on RHEL 8 / CentOS Stream 8

Install Logstash on RHEL 8 / CentOS Stream 8 using the command:

 sudo yum -y install logstash

Once installed, you need to configure Logstash by creating a config at /etc/logstash/conf.d/. The file will have 3 sections i.e the input, filter, and output. All the 3 sections can exist in a single or in 3 different files.

sudo vi /etc/logstash/conf.d/beats.conf

In this guide, we will have all the 3 sections in a single file as below:

input {
  beats {
    port => 5044
  }
}
filter {
  if [type] == "syslog" {
     grok {
        match => { "message" => "%{SYSLOGLINE}" }
  }
     date {
        match => [ "timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
     }
  }
}
output {
  elasticsearch {
    hosts => ["192.168.205.8:9200"]
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
  }
}

Start and enable Logstash.

sudo systemctl start logstash
sudo systemctl enable logstash

5. Install and Configure Kibana on RHEL 8 / CentOS 8

Now we want to install Kibana, the visualization tool in the ELK stack. Install Kibana using the command:

sudo yum -y install kibana

Once installed, configure Kibana.

sudo vi /etc/kibana/kibana.yml

In the opened file, make the below adjustments:

# line 11 : uncomment and change (listen all)
server.host: "0.0.0.0"
# line 32 : uncomment and change (specify own hostname)
server.name: "node1"
# line 43 : uncomment and change if you need
# set if elasticsearch and Kibana are running on different Host
elasticsearch.hosts: ["http://192.168.205.8:9200"]

Start and enable Kibana.

sudo systemctl enable --now kibana

Allow ports 5601 and 5044 through the firewall.

sudo firewall-cmd --add-port=5601/tcp --permanent
sudo firewall-cmd --add-port=5044/tcp --permanent
sudo firewall-cmd --reload

6. Install and Configure Filebeat on RHEL 8 / CentOS 8

At this point, the Kibana dashboard is available on port 5601. We want to install and configure Filebeat to help collect logs and send them to Logstash which transforms them into a format easily understood by Elasticsearch. Filebeat can be installed on the same or client machine from which you want to collect the logs.

Install Filebeat on RHEL 8 / CentOS Stream 8 using the command:

sudo dnf -y install filebeat

Once installed, proceed and configure it to send logs to Logstash as below:

sudo vi /etc/filebeat/filebeat.yml

In the opened file, make the below configurations.

#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
  # Array of hosts to connect to.
  #hosts: ["localhost:9200"]

.....
#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["192.168.205.8:5044"]
....

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/messages
    #- c:programdataelasticsearchlogs*

Start and enable Filebeat.

sudo systemctl enable --now filebeat

7. Access the Kibana Web Interface.

The Kibana web interface can be accessed using the URL http://IP_Address:5601.

Now enable the Filebeat module on the client machine. Begin by listing modules:

$ sudo filebeat modules list
Enabled:

Disabled:
activemq
apache
auditd
aws
awsfargate
azure
barracuda
bluecoat
cef
checkpoint
cisco
coredns
....

Now enable the desired module; Logstash as below:

$ sudo filebeat modules enable logstash
Enabled logstash

Once enabled, load the index template to Elasticsearch and dashboards to Kibana as below.

$ sudo filebeat setup -E output.logstash.enabled=false -E output.elasticsearch.hosts=['192.168.205.8:9200'] -E setup.kibana.host=192.168.205.8:5601
Overwriting ILM policy is disabled. Set `setup.ilm.overwrite: true` for enabling.

Index setup finished.
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards
Loaded Ingest pipelines

Now on the Kibana interface under discover, you will have Filebeat as below.

You can also visualize the data using desired chart/graph.

The end!

Closing Thoughts.

At this point, we are safe to assume that the Elastic Stack 8 (ELK 8) setup on RHEL 8 / CentOS Stream 8 is working perfectly. We can now collect logs and send them to Logstash or Elasticsearch. I hope this was significant.

Related posts:

• How To Install ELK Stack on CentOS / Fedora
• Forward Kubernetes Logs to Elasticsearch (ELK) using Fluentbit
• Analyze Nginx/Apache Log Files using SEO Log File Analyser