Learn how to – How To Monitor Linux Server Uptime with Prometheus

Learn how to – How To Monitor Linux Server Uptime with Prometheus.

“Some fish love to swim upstream. Some people love to overcome challenges.”
― Amit Ray

Ensuring that your infrastructure is kept at the high SLA levels that companies and organizations always demand, then you will have no choice than to find out how you can measure, observe and submit reports that capture the truth about systems performance. There is no shortcut about this when those cold and chilly questions are asked. You can either choose to present a blank perplexed face or one filled with warmth, confidence and furnished with sufficiently concrete reports.

In order to achieve the latter, this guide will walk us through how we can capture Linux Server systems uptime. And to accomplish that, we have to work with a tool known as Prometheus. We know you have heard of Prometheus but with an excuse of re-enforcing it, allow us to introduce it once more. Prometheus is a monitoring tool designed for recording real-time metrics in a time-series database. It is an open-source software project, written in Go and the Prometheus metrics are collected using HTTP pulls, allowing for higher performance and scalability.

In this tutorial, as the title suggests we will discuss how you can monitor Linux server uptime using Prometheus. We shall install it and use its robust capabilities to fetch the data of our interest. We shall therefore begin.

Step 1: Prepare your server

Depending on the environment that your servers are in, you should update and install all the essential packages that you usually use such as text editor (vim or nano), wget or curl and others that you cannot do without.

####For Ubuntu####

sudo apt update && sudo apt upgrade
sudo apt install vim wget curl -y

####For CentOS####

sudo yum update
sudo yum install vim wget curl -y

Step 2: Install Prometheus

Since Prometheus is the centre stage of this guide, we shall proceed to set it up and ensure it is ready to serve us. Lucky for us, we have comprehensive guides for installation. Whether you are on CentOS or Debian or Ubuntu, you are sorted. Check below links to get Prometheus installed.

Install Prometheus Server on Ubuntu | Debian
How To Install Prometheus on RHEL | CentOS 8

Install node_exporter

First, let us create Node Exporter System User which will facilitate the safety of running it. Create such user as follows:

sudo useradd -M -r -s /bin/false node_exporter

After that, download and Install Node Exporter as follows. Point your browser to Prometheus downloads page and fetch the latest version of Node Exporter.

curl -s https://api.github.com/repos/prometheus/node_exporter/releases/latest| grep browser_download_url| grep linux-amd64| cut -d '"' -f 4| wget -qi -

To get Node Exporter installed, we will simply copy the node_exporter binary from the archive folder to /usr/local/bin which is in PATH environment variable.

tar -xvf node_exporter*.tar.gz
cd  node_exporter*/
sudo cp node_exporter /usr/local/bin

Create node_exporter service.

cat << EOF | sudo tee /etc/systemd/system/node_exporter.service
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=default.target
EOF

Reload Systemd and start the service.

sudo systemctl daemon-reload
sudo systemctl start node_exporter
sudo systemctl enable node_exporter

After that, in case you have a firewall running, you will have to allow port 9100 that node_exporter uses by default.

##On Ubuntu

sudo ufw allow 9100

##On CentOS

sudo firewall-cmd --add-port=9100/tcp --permanent && sudo firewall-cmd --reload

Step 3: Add a target to Monitor via Prometheus

On the target node, install node_exporter only

Install node_exporter

Create Node Exporter System User

sudo useradd -M -r -s /bin/false node_exporter

After that, download and Install Node Exporter as follows. Point your browser to Prometheus downloads page and fetch the latest version of Node Exporter.

curl -s https://api.github.com/repos/prometheus/node_exporter/releases/latest| grep browser_download_url| grep linux-amd64| cut -d '"' -f 4| wget -qi -

To get Node Exporter installed, we will simply copy the node_exporter binary from the archive folder to /usr/local/bin.

tar -xvf node_exporter*.tar.gz
cd  node_exporter*/
sudo cp node_exporter /usr/local/bin

Validate installation by checking the software version:

$ node_exporter --version
node_exporter, version 1.0.1 (branch: HEAD, revision: 3715be6ae899f2a9b9dbfd9c39f3e09a7bd4559f)
  build user:       root@1f76dbbcfa55
  build date:       20200616-12:44:12
  go version:       go1.14.4

Create node_exporter service.

cat << EOF | sudo tee /etc/systemd/system/node_exporter.service
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=default.target
EOF

Reload systemd and start the service.

sudo systemctl daemon-reload
sudo systemctl start node_exporter
sudo systemctl enable node_exporter

After that, in case you have a firewall running, you will have to allow port 9100 that node_exporter uses by default.

##On Ubuntu

sudo ufw allow 9100

##On CentOS

sudo firewall-cmd --add-port=9100/tcp --permanent && sudo firewall-cmd --reload

Update Prometheus Server configurations

After node_exporter has been installed, we will have to update our Prometheus server configuration file to get the target of the new node/server.

Add the sections on the end of the configuration file as follows

$ sudo vim /etc/prometheus/prometheus.yml

# This global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['localhost:9090']

  - job_name: node
    static_configs:
      - targets: ['172.28.204.30:9100']

Then restart prometheus

sudo systemctl restart prometheus

When you go back to the browser, you should see a page like below with the new node on the list

Step 4: Add Blackbox exporter

The blackbox exporter allows blackbox probing of endpoints over HTTP, HTTPS, DNS, TCP and ICMP. It runs independently then Prometheus binds to it to, issues requests to the endpoints exposed by blackbox and does an action as configured.

First, let us create blackbox Exporter System User which will facilitate the safety of running it. Create such user as follows:

sudo useradd -M -r -s /bin/false blackbox

Install Blackbox exporter

We are going to fetch and install Blackbox exporter from its latest releases page

cd ~
curl -s https://api.github.com/repos/prometheus/blackbox_exporter/releases/latest | grep browser_download_url | grep linux-amd64 | cut -d '"' -f 4 | wget -qi -

Extract the downloaded file an copy the binaries to a directory in PATH environment variable

tar xvzf blackbox_exporter-*.linux-amd64.tar.gz
cd blackbox_exporter-*.linux-amd64
sudo cp blackbox_exporter /usr/local/bin

Confirm installation:

$ blackbox_exporter --version
blackbox_exporter, version 0.18.0 (branch: HEAD, revision: 60c86e6ce5a1111f7958b06ae7a08222bb6ec839)
  build user:       root@53d72328d93f
  build date:       20201012-09:46:31
  go version:       go1.15.2

Ensure blackbox can read the binaries

$ sudo chown blackbox:blackbox /usr/local/bin/blackbox_exporter

Before getting out of the same directory, we shall create blackbox config files and make sure that blackbox user has the requisite rights to read the files as shown below

cd ~/blackbox_exporter-*.linux-amd64
sudo mkdir -p /etc/blackbox
sudo cp blackbox.yml /etc/blackbox
sudo chown -R blackbox:blackbox /etc/blackbox/*

After copying the binaries to the directory in PATH, we shall proceed to create a Systemd service file for it as follows:

$ sudo vim /etc/systemd/system/blackbox.service

[Unit]
Description=Blackbox Exporter Service
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=blackbox
Group=blackbox
ExecStart=/usr/local/bin/blackbox_exporter 
  --config.file=/etc/blackbox/blackbox.yml 
  --web.listen-address=":9115"

Restart=always

[Install]
WantedBy=multi-user.target

As you can see from the service file, blackbox will be listening on port 9115. As you can guess we will need to allow this port on the firewall in case you have one running as follows

##On Ubuntu

sudo ufw allow 9115

##On CentOS

sudo firewall-cmd --add-port=9115/tcp --permanent && sudo firewall-cmd --reload

Adding a service file requires a daemon-reload then we can start our blackbox service. Execute the commands below to get our blackbox exporter purring.

sudo systemctl daemon-reload
sudo systemctl enable blackbox
sudo systemctl start blackbox

You can then check the status to confirm that the exporter is trully running:

$ sudo systemctl status blackbox

● blackbox.service - Blackbox Exporter Service
     Loaded: loaded (/etc/systemd/system/blackbox.service; disabled; vendor preset: enabled)
     Active: active (running) since Thu 2020-12-03 08:13:03 UTC; 4min 20s ago
   Main PID: 2161 (blackbox_export)
      Tasks: 7 (limit: 2204)
     Memory: 2.2M
     CGroup: /system.slice/blackbox.service
             └─2161 /usr/local/bin/blackbox_exporter --config.file=/etc/blackbox/blackbox.yml --web.listen-address=:9115 

Once we are at this point, we should be able to access blackbox exporter web interface as shared on the screenshot below after pointing the browser to “http://ip-or-fqdn:9115“

Step 5: Monitoring endpoints using Blackbox Exporter

Now that we have Blackbox Exporter installed and running, it is time to take advantage of the gems it provides to make our lives easier. In case you have any http or https endpoints you would wish to monitor, then Blackbox Exporter comes with modules to help you do that. Consider modules to be the soldiers probing for HTTP, HTTPS, DNS, TCP and ICMP on behalf of the exporter.

We are going to monitor simple endpoints that exist in the default blackbox exporter configuration file by binding them to our prometheus server as an example. The default blackbox exporter configuration file comes with pre-defined modules that we can use to bind to our prometheus server. You can view the modules in the “/etc/blackbox/blackbox.yml” file where you installed blackbox.

Bind Blackbox Exporter to Prometheus

On your Prometheus server, add the following configuration at the end of the file that will read the modules on the Blackbox Exporter.

$ sudo vim /etc/prometheus/prometheus.yml

  # any [job_name]
  - job_name: 'ICMP_Probe_Blackbox'
    metrics_path: /probe
    params:
      module: [icmp]
    static_configs:
      - targets:
        # The hostname or IP address of the host you are targeting
        - 172.28.204.30
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        # Where Blackbox exporter was installed plust port
        replacement: 172.28.204.30:9115

  # Using the [ssh_banner] module
  - job_name: 'SSH_Probe_Blackbox'
    metrics_path: /probe
    params:
      module: [ssh_banner]
    static_configs:
      - targets:
        # The host you are targeting plus ssh port
        - 172.28.204.30:22
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        # Where Blackbox exporter was installed plust port
        replacement: 172.28.204.30:9115

  # Using the [http_2xx] module
  - job_name: 'Blackbox_tcp'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
        # The http host you are targeting
        - http://172.28.204.30
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        #Where Blackbox exporter was installed plust port
        replacement: 172.28.204.30:9115

Then restart prometheus

sudo systemctl restart prometheus

When we head over to Prometheus web interface and refresh, we should see all of the probes added as shared in the screenshot below.

Step 6: Viewing Probes

We can view the status of our probes on your Prometheus interface. To do this, click on the “Graph” tab then on the search area written “Expression“, type probe and should see many suggestions popping up.

We shall pick probe_success as an example then click on “Execute“

After picking probe_success, click on the Graph tab below it as shown below:

You should see the probes of success as illustrated

Step 7: Visualize it all on Grafana

Grafana provides visual-worthy features that we cannot dare forget to add to this monitoring ecosystem. In case you have no Grafana installed, we have amazing guides to help you get started quick. They are shared below for you:

How To Install Grafana on CentOS 8 / RHEL 8
Install Grafana on Fedora
Setup Grafana 7 on Ubuntu | Debian Linux

Once you have Grafana installed, make sure you can reach the IP and port of your Prometheus server then proceed to add it as your datasource as illustrated next. Login then:

Move your mouse over the settings gear icon then choose “Data Sources“

Then click on “Add Datasource“

Select Prometheus

Enter the required details such as where Promethueus is running at and so on. You can give it a name you prefer as well.

After you are done scroll down and click “Save & Test“. If all goes well, you should see a good message on Green as shown below

Once your datasource is added, we need to add a dashboard that will summarize what we need to view/monitor. Move your mouse over the + icon and click on “Import“

Enter dashboard by Sparanoid which has an id of 7587. Click on “Import”

A new page will shown up with details of the creator of the dashboard. Select you prometheus datasource then click on “Import“

The moment you click on Import, the dashboard in its splendour should show up as shared below

And we should be done with our monitoring setup

Final Words

Prometheus is versatile and allows you to do much more than scraping metrics for visualization. It can be your monitoring server to get status of sshd, http and https endpoints, your MySQL port and much more. Next place to be is Prometheus documentation, specifically QUERYING PROMETHEUS

Monitor Apache Kafka with Prometheus and Grafana

Monitor Linux Server Performance with Prometheus and Grafana in 5 minutes