Learn how to – How To Monitor Etcd Cluster with Prometheus and Grafana

Learn how to – How To Monitor Etcd Cluster with Prometheus and Grafana.

Monitoring is the next imperative duty after setting up any service used in Production. Having proper monitoring for your Etcd cluster ensures you can respond to issues quickly and know when it’s time to scale. There are a plethora of tools that can be used to monitor an etcd cluster. This guide will discuss the use of Prometheus and Grafana Dashboards, with optional alerting that can be configured.

Each etcd server provides local monitoring information on its client port through http endpoints. The monitoring data is useful for both system health checking and cluster debugging. Test by running:

$ curl -L http://localhost:2379/metrics

Before you proceed, I expect you to have a ready Etcd cluster on a Linux server. We have a detailed guide on how to setup an Etcd cluster on Linux -CentOS/Ubuntu e.t.c. With a running cluster, you can proceed to setup Prometheus and Grafana to be used in monitoring.

Step 1: Install Grafana

You need Grafana Data visualization & Monitoring tool installed on a Linux system. We have a number of guides you can refer to.

Install Grafana on Ubuntu / CentOS / Fedora.

Step 2: Install Prometheus

Prometheus and Grafana can co-exist on same server. For the monitoring to work, the Prometheus server should be able to access Etcd nodes on the service ports.

Check our recent guide which covers the installation of Prometheus on CentOS & Ubuntu Linux systems.

Step 3: Configure Prometheus

We need to set Prometheus’s scraper to target the etcd cluster endpoints:

sudo vim /etc/prometheus/prometheus.yml

Add like below.

- job_name: 'etcd'
    static_configs:
    - targets: ['192.168.18.9:2379','192.168.18.10:2379','192.168.18.11:2379'']

Replace the IP addresses specified with your Etcd cluster node’s IP addresses. Restart Prometheus service after adding the configuration lines to the file.

sudo systemctl restart prometheus

The targets should show on Prometheus dashboard Status > Targets section.

Step 4: Add default etcd dashboard

You can start with the default etcd dashboard for Grafana then customize it to your taste. Check Etcd monitoring guide for more details.

Add data source to Grafana.

Configuration > Data Sources > Add data source > Prometheus

Example:

Once added, import Dashboard default etcd dashboard or search Grafana for etcd dashboards submitted by the community.

Select Prometheus data source added.

Customize the dashboards to display your desired metrics on grafana.

And you have Etcd cluster on Grafana.

Similar guides:

• Monitoring Ceph Cluster with Prometheus and Grafana
• Monitoring Apache Web server with Prometheus and Grafana
• Monitor Linux Server Performance with Prometheus and Grafana in 5 minutes
• How to Monitor BIND DNS server with Prometheus and Grafana
• How to Monitor Redis Server with Prometheus and Grafana in 5 minutes
• Monitoring MySQL / MariaDB with Prometheus in five minutes