Learn how to – Deploying Prometheus on EKS Kubernetes Cluster.
Prometheus is an open-source monitoring system with a dimensional data model, flexible query language, efficient time series database and modern alerting approach. Prometheus can be installed as standalone service in a Linux machine or deployed in a Kubernetes cluster. In this guide we will walk you through the installation of Prometheus on an EKS Cluster deployed in AWS Cloud.
The main features of Prometheus are:
- a multi-dimensional data model with time series data identified by metric name and key/value pairs
- PromQL, a flexible query language to leverage this dimensionality
- no reliance on distributed storage; single server nodes are autonomous
- time series collection happens via a pull model over HTTP
- pushing time series is supported via an intermediary gateway
- targets are discovered via service discovery or static configuration
- multiple modes of graphing and dashboarding support
Deploying Prometheus on EKS Kubernetes Cluster
You should have a running EKS Cluster before you can use this guide with kubectl installed and able to interact with the API server.
List available clusters.
$ eksctl get cluster
NAME REGION
prod-eks-cluster eu-west-1Step 1: Configure kubectl
To obtain cluster credentials at any point in time from an EKS cluster deployed with eksctl, run:
$ eksctl utils write-kubeconfig --cluster=<name> [--kubeconfig=<path>][--set-kubeconfig-context=<bool>]See below example:
# Create default kubeconfig directory - Can be custom
--- Linux ---
$ mkdir /home/$USER/.kube
--- macOS ---
$ mkdir /Users/$USER/.kube
# Obtain cluster credentials
--- Linux ---
$ s write-kubeconfig --cluster=prod-eks-cluster --kubeconfig=/home/$USER/.kube/config
--- macOS ---
$ eksctl utils write-kubeconfig --cluster=prod-eks-cluster --kubeconfig=/Users/$USER/.kube/config
Expected command execution output:
[ℹ] eksctl version 0.25.0
[ℹ] using region eu-west-1
[✔] saved kubeconfig as "/Users/jkmutai/.kube/config"Confirm if kubectl command is working as expected.
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-192-168-138-244.eu-west-1.compute.internal Ready <none> 7d21h v1.17.9-eks-4c6976
ip-192-168-176-247.eu-west-1.compute.internal Ready <none> 7d21h v1.17.9-eks-4c6976Step 2: Install Kubernetes Metrics Server
The Kubernetes API server exposes a number of metrics that are useful for monitoring and analysis. These metrics are exposed internally through a metrics endpoint that refers to the /metrics HTTP API.
Install Metrics Server with the following guide:
Install Kubernetes Metrics Server on Amazon EKS Cluster
You can view cluster raw metrics by running the following command:
$ kubectl get --raw /metricsStep 3: Install Helm
The Helm package manager for Kubernetes helps you install and manage applications on your Kubernetes cluster. We’ll be using help to deploy Prometheus on EKS.
These are the commands you’ll use to install helm.
--- Linux ---
curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 > get_helm.sh
chmod 700 get_helm.sh
./get_helm.sh
--- macOS ---
$ brew install helm
--- Windows ---
$ choco install kubernetes-helm
You can query installed version using the command:
$ helm version
version.BuildInfo{Version:"v3.2.4", GitCommit:"0ad800ef43d3b826f31a5ad8dfbb4fe05d143688", GitTreeState:"dirty", GoVersion:"go1.14.3"}Step 4: Deploying Prometheus on EKS Kubernetes Cluster
Prometheus can be installed on Kubernetes cluster using Operator or with helm. We’ll use Helm V3 we just installed to deploy prometheus in an EKS cluster.
First create a monitoring namespace.
$ kubectl create namespace monitoring
namespace/monitoring createdPrometheus needs a way to persist metrics data for historical reference. We’ll use EBS which is provisioned with gp2 storage class.
$ kubectl get sc
kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
efs-sc efs.csi.aws.com Delete Immediate false 2d23h
gp2 (default) kubernetes.io/aws-ebs Delete WaitForFirstConsumer false 7d21h
Add chart repository:
$ helm repo add stable https://kubernetes-charts.storage.googleapis.com
"stable" has been added to your repositoriesDeploy Prometheus using Helm.
helm install prometheus stable/prometheus
--namespace monitoring
--set alertmanager.persistentVolume.storageClass="gp2",server.persistentVolume.storageClass="gp2"Command execution output:
NAME: prometheus
LAST DEPLOYED: Wed Aug 19 22:01:16 2020
NAMESPACE: monitoring
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The Prometheus server can be accessed via port 80 on the following DNS name from within your cluster:
prometheus-server.monitoring.svc.cluster.localConfirm PV and PVC are created.
$ kubectl get pv -n monitoring
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-7a1d891c-5b3f-4d28-91b5-5f1d1590daf9 8Gi RWO Delete Bound monitoring/prometheus-server gp2 11m
pvc-7d45951f-a8c5-4fb1-b149-747dccaceecc 2Gi RWO Delete Bound monitoring/prometheus-alertmanager gp2 11m
$ kubectl get pvc -n monitoring
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
prometheus-alertmanager Bound pvc-7d45951f-a8c5-4fb1-b149-747dccaceecc 2Gi RWO gp2 11m
prometheus-server Bound pvc-7a1d891c-5b3f-4d28-91b5-5f1d1590daf9 8Gi RWO gp2 11mStep 4: Access Prometheus on EKS Kubernetes Cluster
After installation query all resources in the monitoring namespace:
$ kubectl get all -n monitoring
NAME READY STATUS RESTARTS AGE
pod/prometheus-alertmanager-d47577c4b-kz9td 2/2 Running 0 98s
pod/prometheus-kube-state-metrics-6df5d44568-fzfwn 1/1 Running 0 98s
pod/prometheus-node-exporter-p6qk7 1/1 Running 0 99s
pod/prometheus-node-exporter-stsjk 1/1 Running 0 99s
pod/prometheus-pushgateway-57c97d878d-hmnwr 1/1 Running 0 98s
pod/prometheus-server-559c49b4ff-42k46 2/2 Running 0 98s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/prometheus-alertmanager ClusterIP 10.100.136.125 <none> 80/TCP 99s
service/prometheus-kube-state-metrics ClusterIP 10.100.169.64 <none> 8080/TCP 99s
service/prometheus-node-exporter ClusterIP None <none> 9100/TCP 99s
service/prometheus-pushgateway ClusterIP 10.100.71.173 <none> 9091/TCP 99s
service/prometheus-server ClusterIP 10.100.138.66 <none> 80/TCP 99s
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/prometheus-node-exporter 2 2 2 2 2 <none> 99s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/prometheus-alertmanager 1/1 1 1 99s
deployment.apps/prometheus-kube-state-metrics 1/1 1 1 99s
deployment.apps/prometheus-pushgateway 1/1 1 1 99s
deployment.apps/prometheus-server 1/1 1 1 99s
NAME DESIRED CURRENT READY AGE
replicaset.apps/prometheus-alertmanager-d47577c4b 1 1 1 99s
replicaset.apps/prometheus-kube-state-metrics-6df5d44568 1 1 1 99s
replicaset.apps/prometheus-pushgateway-57c97d878d 1 1 1 99s
replicaset.apps/prometheus-server-559c49b4ff 1 1 1 99sGet the Prometheus server URL by running these commands in the same shell:
export POD_NAME=$(kubectl get pods --namespace monitoring -l "app=prometheus,component=server" -o jsonpath="{.items[0].metadata.name}")Use Kubernetes port forwarding feature to access Prometheus Server.
$ kubectl --namespace monitoring port-forward $POD_NAME 9090
Forwarding from 127.0.0.1:9090 -> 9090
Forwarding from [::1]:9090 -> 9090Access 127.0.0.1:9090 from your local machine’s browser.
Similar commands are used to access Alertmanager UI
export POD_NAME=$(kubectl get pods --namespace monitoring -l "app=prometheus,component=alertmanager" -o jsonpath="{.items[0].metadata.name}")
kubectl --namespace monitoring port-forward $POD_NAME 9093If you want to access Prometheus Server and Alertmanager with Node Ports, edit the services and set NodePort.
Next place to be is Prometheus documentation, specifically QUERYING PROMETHEUS
More guides:
Using Horizontal Pod Autoscaler on Kubernetes EKS Cluster
Enable CloudWatch logging in EKS Kubernetes Cluster
EKS Kubernetes Persistent Storage with EFS Storage Service
Leave a Reply
You must be logged in to post a comment.