Learn how to - Setup GlusterFS Storage With Heketi on CentOS 8 / CentOS 7

Felix Williams Uncategorized May 30, 2023

Learn how to – Setup GlusterFS Storage With Heketi on CentOS 8 / CentOS 7.

In this guide, you’ll learn to install and configure GlusterFS Storage on CentOS 8 / CentOS 7 with Heketi. GlusterFS is a software defined, scale-out storage solution designed to provide affordable and flexible storage for unstructured data. GlusterFS allows you to unify infrastructure and data storage while improving availability performance and data manageability.

GlusterFS Storage can be deployed in the private cloud or datacenter or in your in-Premise datacenter. This is done purely on commodity servers and storage hardware resulting in a powerful, massively scalable, and highly available NAS environment.

Heketi

Heketi provides a RESTful management interface which can be used to manage the lifecycle of GlusterFS Storage volumes. This allows for easy integration of GlusterFS with cloud services like OpenShift, OpenStack Manila and Kubernetes for dynamic volumes provisioning.

Heketi will automatically determine the location for bricks across the cluster, making sure to place bricks and its replicas across different failure domains.

Environment Setup

Our setup of GlusterFS on CentOS 8 / CentOS 7 systems will comprise of below.

CentOS 8 / CentOS 8 Linux servers
GlusterFS 6 software release
Three GlusterFS Servers
Each server has three disks (@10GB)
DNS resolution configured – You can use /etc/hosts file if you don’t have DNS server
User account with sudo or root user access
Heketi will be installed in one of the GlusterFS nodes.

Under the /etc/hosts file of each server, I have:

$ sudo vim /etc/hosts
10.10.1.168 gluster01
10.10.1.179 gluster02
10.10.1.64  gluster03

Step 1: Update all servers

Ensure all servers that will be part of the GlusterFS storage cluster are updated.

sudo yum -y update

Since there may be kernel updates, I recommend you reboot your system.

sudo reboot

Step 2: Configure NTP time synchronization

You need to synchronize time across all GlusterFS Storage servers using the Network Time Protocol (NTP) or Chrony daemon. Refer to our guide below.

Setup Chrony Time synchronization on CentOS

Step 3: Add GlusterFS repository

Download GlusterFS repository on all servers. We’ll do GlusterFS 6 in this setup since it is the latest stable release.

CentOS 8:

sudo yum -y install wget vim
sudo yum install centos-release-gluster -y

CentOS 7:

sudo yum -y install wget vim
sudo yum -y install centos-release-gluster9

Once you’ve added the repository, update your YUM index.

sudo yum makecache

Step 4: Install GlusterFS on CentOS 8 / CentOS 7

Installation of GlusterFS on CentOS 8 differs from CentOS 7 installation.

Install GlusterFS on CentOS 8

Enable PowerTools repository

sudo dnf -y install dnf-utils
sudo yum-config-manager --enable powertools
sudo dnf -y install glusterfs-server

Install GlusterFS on CentOS 7

Run the following commands on all nodes to install latest GlusterFS on CentOS 7.

sudo yum -y install glusterfs-server

Confirm installed package version.

### CentOS 8 ###
$ rpm -qi glusterfs-server
Name        : glusterfs-server
Version     : 10.4
Release     : 1.el8s
Architecture: x86_64
Install Date: Mon 05 Jun 2023 10:14:04 PM UTC
Group       : Unspecified
Size        : 6204643
License     : GPLv2 or LGPLv3+
Signature   : RSA/SHA256, Wed 26 Apr 2023 07:28:14 PM UTC, Key ID d4a2e50be451e5b5
Source RPM  : glusterfs-10.4-1.el8s.src.rpm
Build Date  : Wed 26 Apr 2023 09:34:08 AM UTC
Build Host  : cody-n11.rdu2.centos.org
...

### CentOS 7 ###
$ rpm -qi glusterfs-server
Name        : glusterfs-server
Version     : 9.6
Release     : 1.el7
Architecture: x86_64
Install Date: Mon 05 Jun 2023 10:13:29 PM UTC
Group       : Unspecified
Size        : 6230232
License     : GPLv2 or LGPLv3+
Signature   : RSA/SHA256, Fri 19 Aug 2022 09:37:14 AM UTC, Key ID d4a2e50be451e5b5
Source RPM  : glusterfs-9.6-1.el7.src.rpm
Build Date  : Fri 19 Aug 2022 08:03:53 AM UTC
Build Host  : bcb1620d26694117892a9bccb477399c
...

You can also use the gluster command to check version.

$ gluster --version 
glusterfs 10.4
Repository revision: git://git.gluster.org/glusterfs.git
Copyright (c) 2006-2016 Red Hat, Inc. <https://www.gluster.org/>
GlusterFS comes with ABSOLUTELY NO WARRANTY.

$ glusterfsd --version

Step 5: Start GlusterFS Service

After installation of GlusterFS Service on CentOS 8 / CentOS 7, start and enable the service.

sudo systemctl enable --now glusterd.service

Load all Kernel modules that will be required by Heketi.

for i in dm_snapshot dm_mirror dm_thin_pool; do
  sudo modprobe $i
done

If you have an active firewalld service, allow ports used by GlusterFS.

sudo firewall-cmd --add-service=glusterfs --permanent 
sudo firewall-cmd --reload

Check service status on all nodes.

$ systemctl status glusterd glusterd glusterd
● glusterd.service - GlusterFS, a clustered file-system server
   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2022-10-26 20:26:30 UTC; 15s ago
     Docs: man:glusterd(8)
  Process: 3523 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 3524 (glusterd)
    Tasks: 8 (limit: 11061)
   Memory: 13.8M
   CGroup: /system.slice/glusterd.service
           └─3524 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO

Oct 26 20:26:29 centos8.mylab.io systemd[1]: Starting GlusterFS, a clustered file-system server...
Oct 26 20:26:30 centos8.mylab.io systemd[1]: Started GlusterFS, a clustered file-system server.

● glusterd.service - GlusterFS, a clustered file-system server
   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2022-10-26 20:26:30 UTC; 15s ago
     Docs: man:glusterd(8)
  Process: 3523 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 3524 (glusterd)
    Tasks: 8 (limit: 11061)
   Memory: 13.8M
   CGroup: /system.slice/glusterd.service
           └─3524 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO

Oct 26 20:26:29 centos8.mylab.io systemd[1]: Starting GlusterFS, a clustered file-system server...
Oct 26 20:26:30 centos8.mylab.io systemd[1]: Started GlusterFS, a clustered file-system server.

● glusterd.service - GlusterFS, a clustered file-system server
   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2022-10-26 20:26:30 UTC; 15s ago
     Docs: man:glusterd(8)
  Process: 3523 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 3524 (glusterd)
    Tasks: 8 (limit: 11061)
   Memory: 13.8M
   CGroup: /system.slice/glusterd.service
           └─3524 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO

Oct 26 20:26:29 centos8.mylab.io systemd[1]: Starting GlusterFS, a clustered file-system server...
Oct 26 20:26:30 centos8.mylab.io systemd[1]: Started GlusterFS, a clustered file-system server.

Probe other nodes in the cluster

[root@gluster01 ~]# gluster peer probe gluster02
peer probe: success. 

[root@gluster01 ~]# gluster peer probe gluster03
peer probe: success. 

[root@gluster01 ~]# gluster peer status
Number of Peers: 2

Hostname: gluster02
Uuid: ebfdf84f-3d66-4f98-93df-a6442b5466ed
State: Peer in Cluster (Connected)

Hostname: gluster03
Uuid: 98547ab1-9565-4f71-928c-8e4e13eb61c3
State: Peer in Cluster (Connected)

Step 6: Install Heketi on one of the nodes

I’ll use gluster01 node to run Heketi service. Download the latest arhives of Heketi server and client from Github releases page.

wget https://github.com/heketi/heketi/releases/download/v10.4.0/heketi-v10.4.0-release-10.linux.amd64.tar.gz

Extract downloaded heketi archives.

for i in `ls | grep heketi | grep .tar.gz`; do tar xvf $i; done

Copy the heketi & heketi-cli binary packages.

sudo cp heketi/{heketi,heketi-cli} /usr/local/bin

Confirm they are available in your PATH

$ heketi --version
Heketi v10.4.0-release-10 (using go: go1.15.14)

$ heketi-cli --version
heketi-cli v10.4.0-release-10

Step 7: Configure Heketi Server

Add heketi system user.

sudo groupadd --system heketi
sudo useradd -s /sbin/nologin --system -g heketi heketi

Create heketi configurations and data paths.

sudo mkdir -p /var/lib/heketi /etc/heketi /var/log/heketi

Copy heketi configuration file to /etc/heketi directory.

sudo cp heketi/heketi.json /etc/heketi

Edit the Heketi configuration file

sudo vim /etc/heketi/heketi.json

Set service port:

"port": "8080"

Set admin and use secrets.

"_jwt": "Private keys for access",
  "jwt": {
    "_admin": "Admin has access to all APIs",
    "admin": {
      "key": "ivd7dfORN7QNeKVO"
    },
    "_user": "User only has access to /volumes endpoint",
    "user": {
      "key": "gZPgdZ8NtBNj6jfp"
    }
  },

Configure glusterfs executor

_sshexec_comment": "SSH username and private key file information",
    "sshexec": {
      "keyfile": "/etc/heketi/heketi_key",
      "user": "root",
      "port": "22",
      "fstab": "/etc/fstab",
......
},

If you use a user other than root, ensure it has passwordless sudo privilege escalation.

Confirm database path is set properly

"_db_comment": "Database file name",
    "db": "/var/lib/heketi/heketi.db",
},

Generate Heketi SSH keys

sudo ssh-keygen -f /etc/heketi/heketi_key -t rsa -N ''
sudo chown heketi:heketi /etc/heketi/heketi_key*

Copy generated public key to all GlusterFS nodes

for i in gluster01 gluster02 gluster03; do
  ssh-copy-id -i /etc/heketi/heketi_key.pub root@$i
done

Alternatively, you can cat the contents of /etc/heketi/heketi_key.pub and add to each server ~/.ssh/authorized_keys

Confirm you can access the GlusterFS nodes with the Heketi private key:

$ ssh -i /etc/heketi/heketi_key root@gluster02
The authenticity of host 'gluster02 (10.10.1.179)' can't be established.
ECDSA key fingerprint is SHA256:GXNdsSxmp2O104rPB4RmYsH73nTa5U10cw3LG22sANc.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'gluster02,10.10.1.179' (ECDSA) to the list of known hosts.
Activate the web console with: systemctl enable --now cockpit.socket

Last login: Tue Oct 29 20:11:32 2019 from 10.10.1.168
[root@gluster02 ~]#

Create Systemd Unit file

Create Systemd unit file for Heketi

$ sudo vim /etc/systemd/system/heketi.service
[Unit]
Description=Heketi Server

[Service]
Type=simple
WorkingDirectory=/var/lib/heketi
EnvironmentFile=-/etc/heketi/heketi.env
User=heketi
ExecStart=/usr/local/bin/heketi --config=/etc/heketi/heketi.json
Restart=on-failure
StandardOutput=syslog
StandardError=syslog

[Install]
WantedBy=multi-user.target

Also download sample environment file for Heketi.

sudo wget -O /etc/heketi/heketi.env https://raw.githubusercontent.com/heketi/heketi/master/extras/systemd/heketi.env

Set all directory permissions

sudo chown -R heketi:heketi /var/lib/heketi /var/log/heketi /etc/heketi

Start Heketi service

Disable SELinux

sudo setenforce 0
sudo sed -i 's/^SELINUX=.*/SELINUX=permissive/g' /etc/selinux/config

Then reload Systemd and start Heketi service

sudo systemctl daemon-reload
sudo systemctl enable --now heketi

Confirm the service is running.

$ systemctl status heketi
● heketi.service - Heketi Server
   Loaded: loaded (/etc/systemd/system/heketi.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2019-10-29 20:29:23 EAT; 4s ago
 Main PID: 2166 (heketi)
    Tasks: 5 (limit: 11512)
   Memory: 8.7M
   CGroup: /system.slice/heketi.service
           └─2166 /usr/local/bin/heketi --config=/etc/heketi/heketi.json

Oct 29 20:29:23 gluster01.novalocal heketi[2166]: Heketi v9.0.0
Oct 29 20:29:23 gluster01.novalocal heketi[2166]: [heketi] INFO 2019/10/29 20:29:23 Loaded mock executor
Oct 29 20:29:23 gluster01.novalocal heketi[2166]: [heketi] INFO 2019/10/29 20:29:23 Volumes per cluster limit is set to default value of 1000
Oct 29 20:29:23 gluster01.novalocal heketi[2166]: [heketi] INFO 2019/10/29 20:29:23 Block: Auto Create Block Hosting Volume set to true
Oct 29 20:29:23 gluster01.novalocal heketi[2166]: [heketi] INFO 2019/10/29 20:29:23 Block: New Block Hosting Volume size 500 GB
Oct 29 20:29:23 gluster01.novalocal heketi[2166]: [heketi] INFO 2019/10/29 20:29:23 Block: New Block Hosting Volume Options: group gluster-block
Oct 29 20:29:23 gluster01.novalocal heketi[2166]: [heketi] INFO 2019/10/29 20:29:23 GlusterFS Application Loaded
Oct 29 20:29:23 gluster01.novalocal heketi[2166]: [heketi] INFO 2019/10/29 20:29:23 Started background pending operations cleaner
Oct 29 20:29:23 gluster01.novalocal heketi[2166]: [heketi] INFO 2019/10/29 20:29:23 Started Node Health Cache Monitor
Oct 29 20:29:23 gluster01.novalocal heketi[2166]: Listening on port 8080

Step 8: Create Heketi Topology file

I’ve created an ansible playbook to be used for generating and updating topology file. Editing json file manually can be stressing. This will make scaling easy.

Install Ansible Locally – refer to Ansible Installation of Ansible documentation.

For CentOS:

sudo yum -y install epel-release
sudo yum -y install ansible

For Ubuntu:

sudo apt update
sudo apt install software-properties-common
sudo apt-add-repository --yes --update ppa:ansible/ansible
sudo apt install ansible

Once Ansible is installed, create project folder structure

mkdir -p ~/projects/ansible/roles/heketi/{tasks,templates,defaults}

Create Heketi Topology Jinja2 template

$ vim ~/projects/ansible/roles/heketi/templates/topology.json.j2

{

  "clusters": [

    {

      "nodes": [

      {% if gluster_servers is defined and gluster_servers is iterable %}

      {% for item in gluster_servers %}

        {

          "node": {

            "hostnames": {

              "manage": [

                "{{ item.servername }}"

              ],

              "storage": [

                "{{ item.serverip }}"

              ]

            },

            "zone": {{ item.zone }}

          },

          "devices": [

            "{{ item.disks | list | join (""

Cart

Learn how to – Setup GlusterFS Storage With Heketi on CentOS 8 / CentOS 7