Learn how to – Setup Prometheus and Grafana on Kubernetes using prometheus-operator.
Monitoring Production Kubernetes Cluster(s) is an important and progressive operation for any Cluster Administrator. There are myriad of solutions that fall into the category of Kubernetes monitoring stack, and some of them are Prometheus and Grafana. This guide is created with an intention of guiding Kubernetes users to Setup Prometheus and Grafana on Kubernetes using prometheus-operator.
Prometheus is a full fledged solution that enables Developers and SysAdmins to access advanced metrics capabilities in Kubernetes. The metrics are collected in a time interval of 30 seconds, this is a default settings. The information collected include resources such as Memory, CPU, Disk Performance and Network IO as well as R/W rates. By default the metrics are exposed on your cluster for up to a period of 14 days, but the settings can be adjusted to suit your environment.
Grafana is used for analytics and interactive visualization of metrics that’s collected and stored in Prometheus database. You can create custom charts, graphs, and alerts for Kubernetes cluster, with Prometheus being data source. In this guide we will perform installation of both Prometheus and Grafana on a Kubernetes Cluster. For this setup kubectl configuration is required, with Cluster Admin role binding.
Prometheus Operator
We will be using Prometheus Operator in this installation to deploy Prometheus monitoring stack on Kubernetes. The Prometheus Operator is written to ease the deployment and overall management of Prometheus and its related monitoring components. By using the Operator we simplify and automate Prometheus configuration on any any Kubernetes cluster using Kubernetes custom resources.
The diagram below shows the components of the Kubernetes monitoring that we’ll deploy:
The Operator uses the following custom resource definitions (CRDs) to deploy and configure Prometheus monitoring stack:
- Prometheus – This defines a desired Prometheus deployment on Kubernetes
- Alertmanager – This defines a desired Alertmanager deployment on Kubernetes cluster
- ThanosRuler – This defines Thanos desired Ruler deployment.
- ServiceMonitor – Specifies how groups of Kubernetes services should be monitored
- PodMonitor – Declaratively specifies how group of pods should be monitored
- Probe – Specifies how groups of ingresses or static targets should be monitored
- PrometheusRule – Provides specification of Prometheus alerting desired set. The Operator generates a rule file, which can be used by Prometheus instances.
- AlertmanagerConfig – Declaratively specifies subsections of the Alertmanager configuration, allowing routing of alerts to custom receivers, and setting inhibit rules.
Deploy Prometheus / Grafana Monitoring Stack on Kubernetes
To get a complete an entire monitoring stack we will use kube-prometheus project which includes Prometheus Operator among its components. The kube-prometheus stack is meant for cluster monitoring and is pre-configured to collect metrics from all Kubernetes components, with a default set of dashboards and alerting rules.
You should have kubectl configured and confirmed to be working:
$ kubectl cluster-info
Kubernetes control plane is running at https://192.168.10.12:6443
CoreDNS is running at https://192.168.10.12:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.Step 1: Clone kube-prometheus project
Use git command to clone kube-prometheus project to your local system:
git clone https://github.com/prometheus-operator/kube-prometheus.gitNavigate to the kube-prometheus directory:
cd kube-prometheus
Step 2: Create monitoring namespace, CustomResourceDefinitions & operator pod
Create a namespace and required CustomResourceDefinitions:
kubectl create -f manifests/setupCommand execution results as seen in the terminal screen.
customresourcedefinition.apiextensions.k8s.io/alertmanagerconfigs.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/alertmanagers.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/podmonitors.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/probes.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/prometheuses.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/prometheusrules.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/servicemonitors.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/thanosrulers.monitoring.coreos.com created
namespace/monitoring created
The namespace created with CustomResourceDefinitions is named monitoring:
$ kubectl get ns monitoring
NAME STATUS AGE
monitoring Active 2m41sStep 3: Deploy Prometheus Monitoring Stack on Kubernetes
Once you confirm the Prometheus operator is running you can go ahead and deploy Prometheus monitoring stack.
kubectl create -f manifests/Here is my deployment progress output:
poddisruptionbudget.policy/alertmanager-main created
prometheusrule.monitoring.coreos.com/alertmanager-main-rules created
secret/alertmanager-main created
service/alertmanager-main created
serviceaccount/alertmanager-main created
servicemonitor.monitoring.coreos.com/alertmanager created
clusterrole.rbac.authorization.k8s.io/blackbox-exporter created
clusterrolebinding.rbac.authorization.k8s.io/blackbox-exporter created
configmap/blackbox-exporter-configuration created
deployment.apps/blackbox-exporter created
service/blackbox-exporter created
serviceaccount/blackbox-exporter created
servicemonitor.monitoring.coreos.com/blackbox-exporter created
secret/grafana-datasources created
configmap/grafana-dashboard-alertmanager-overview created
configmap/grafana-dashboard-apiserver created
configmap/grafana-dashboard-cluster-total created
configmap/grafana-dashboard-controller-manager created
configmap/grafana-dashboard-k8s-resources-cluster created
configmap/grafana-dashboard-k8s-resources-namespace created
configmap/grafana-dashboard-k8s-resources-node created
configmap/grafana-dashboard-k8s-resources-pod created
configmap/grafana-dashboard-k8s-resources-workload created
configmap/grafana-dashboard-k8s-resources-workloads-namespace created
configmap/grafana-dashboard-kubelet created
configmap/grafana-dashboard-namespace-by-pod created
configmap/grafana-dashboard-namespace-by-workload created
configmap/grafana-dashboard-node-cluster-rsrc-use created
configmap/grafana-dashboard-node-rsrc-use created
configmap/grafana-dashboard-nodes created
configmap/grafana-dashboard-persistentvolumesusage created
configmap/grafana-dashboard-pod-total created
configmap/grafana-dashboard-prometheus-remote-write created
configmap/grafana-dashboard-prometheus created
configmap/grafana-dashboard-proxy created
configmap/grafana-dashboard-scheduler created
configmap/grafana-dashboard-workload-total created
configmap/grafana-dashboards created
deployment.apps/grafana created
service/grafana created
serviceaccount/grafana created
servicemonitor.monitoring.coreos.com/grafana created
prometheusrule.monitoring.coreos.com/kube-prometheus-rules created
clusterrole.rbac.authorization.k8s.io/kube-state-metrics created
clusterrolebinding.rbac.authorization.k8s.io/kube-state-metrics created
deployment.apps/kube-state-metrics created
prometheusrule.monitoring.coreos.com/kube-state-metrics-rules created
service/kube-state-metrics created
serviceaccount/kube-state-metrics created
servicemonitor.monitoring.coreos.com/kube-state-metrics created
prometheusrule.monitoring.coreos.com/kubernetes-monitoring-rules created
servicemonitor.monitoring.coreos.com/kube-apiserver created
servicemonitor.monitoring.coreos.com/coredns created
servicemonitor.monitoring.coreos.com/kube-controller-manager created
servicemonitor.monitoring.coreos.com/kube-scheduler created
servicemonitor.monitoring.coreos.com/kubelet created
clusterrole.rbac.authorization.k8s.io/node-exporter created
clusterrolebinding.rbac.authorization.k8s.io/node-exporter created
daemonset.apps/node-exporter created
prometheusrule.monitoring.coreos.com/node-exporter-rules created
service/node-exporter created
serviceaccount/node-exporter created
servicemonitor.monitoring.coreos.com/node-exporter created
clusterrole.rbac.authorization.k8s.io/prometheus-adapter created
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-adapter created
clusterrolebinding.rbac.authorization.k8s.io/resource-metrics:system:auth-delegator created
clusterrole.rbac.authorization.k8s.io/resource-metrics-server-resources created
configmap/adapter-config created
deployment.apps/prometheus-adapter created
poddisruptionbudget.policy/prometheus-adapter created
rolebinding.rbac.authorization.k8s.io/resource-metrics-auth-reader created
service/prometheus-adapter created
serviceaccount/prometheus-adapter created
servicemonitor.monitoring.coreos.com/prometheus-adapter created
clusterrole.rbac.authorization.k8s.io/prometheus-k8s created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-k8s created
prometheusrule.monitoring.coreos.com/prometheus-operator-rules created
servicemonitor.monitoring.coreos.com/prometheus-operator created
poddisruptionbudget.policy/prometheus-k8s created
prometheus.monitoring.coreos.com/k8s created
prometheusrule.monitoring.coreos.com/prometheus-k8s-prometheus-rules created
rolebinding.rbac.authorization.k8s.io/prometheus-k8s-config created
rolebinding.rbac.authorization.k8s.io/prometheus-k8s created
rolebinding.rbac.authorization.k8s.io/prometheus-k8s created
rolebinding.rbac.authorization.k8s.io/prometheus-k8s created
role.rbac.authorization.k8s.io/prometheus-k8s-config created
role.rbac.authorization.k8s.io/prometheus-k8s created
role.rbac.authorization.k8s.io/prometheus-k8s created
role.rbac.authorization.k8s.io/prometheus-k8s created
service/prometheus-k8s created
serviceaccount/prometheus-k8s created
servicemonitor.monitoring.coreos.com/prometheus-k8s created
Give it few seconds and the pods should start coming online. This can be checked with the commands below:
$ kubectl get pods -n monitoring -w
NAME READY STATUS RESTARTS AGE
alertmanager-main-0 2/2 Running 0 3m8s
alertmanager-main-1 2/2 Running 1 (2m55s ago) 3m8s
alertmanager-main-2 2/2 Running 1 (2m40s ago) 3m8s
blackbox-exporter-69684688c9-nk66w 3/3 Running 0 6m47s
grafana-7bf8dc45db-q2ndq 1/1 Running 0 6m47s
kube-state-metrics-d75597b45-d9bhk 3/3 Running 0 6m47s
node-exporter-2jzcv 2/2 Running 0 6m47s
node-exporter-5k8pk 2/2 Running 0 6m47s
node-exporter-9852n 2/2 Running 0 6m47s
node-exporter-f5dmp 2/2 Running 0 6m47s
prometheus-adapter-5f68766c85-hjcz9 1/1 Running 0 6m46s
prometheus-adapter-5f68766c85-shjbz 1/1 Running 0 6m46s
prometheus-k8s-0 2/2 Running 0 3m7s
prometheus-k8s-1 2/2 Running 0 3m7s
prometheus-operator-748bb6fccf-b5ppx 2/2 Running 0 6m46sTo list all the services created you’ll run the command:
$ kubectl get svc -n monitoring
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
alertmanager-main ClusterIP 10.100.171.41 <none> 9093/TCP,8080/TCP 7m2s
alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 3m23s
blackbox-exporter ClusterIP 10.108.187.73 <none> 9115/TCP,19115/TCP 7m2s
grafana ClusterIP 10.97.236.243 <none> 3000/TCP 7m2s
kube-state-metrics ClusterIP None <none> 8443/TCP,9443/TCP 7m2s
node-exporter ClusterIP None <none> 9100/TCP 7m2s
prometheus-adapter ClusterIP 10.109.119.234 <none> 443/TCP 7m1s
prometheus-k8s ClusterIP 10.101.253.211 <none> 9090/TCP,8080/TCP 7m1s
prometheus-operated ClusterIP None <none> 9090/TCP 3m22s
prometheus-operator ClusterIP None <none> 8443/TCP 7m1sStep 4: Access Prometheus, Grafana, and Alertmanager dashboards
We now have the monitoring stack deployed, but how can we access the dashboards of Grafana, Prometheus and Alertmanager?. There are two ways to achieve this;
Method 1: Accessing Prometheus UI and Grafana dashboards using kubectl proxy
An easy way to access Prometheus, Grafana, and Alertmanager dashboards is by using kubectl port-forward once all the services are running:
Grafana Dashboard
kubectl --namespace monitoring port-forward svc/grafana 3000Then access Grafana dashboard on your local browser on URL: http://localhost:3000
Default Logins are:
Username: admin
Password: adminYou’re required to change the password on first login:
Prometheus Dashboard
For Prometheus port forwarding run the commands below:
kubectl --namespace monitoring port-forward svc/prometheus-k8s 9090And web console is accessible through the URL: http://localhost:9090
Alert Manager Dashboard
For Dashboard Alert Manager Dashboard:
kubectl --namespace monitoring port-forward svc/alertmanager-main 9093Access URL is http://localhost:9093
Method 2: Accessing Prometheus UI and Grafana dashboard using NodePort / LoadBalancer
To access Prometheus, Grafana, and Alertmanager dashboards using one of the worker nodes IP address and a port you’ve to edit the services and set the type to NodePort.
You need a Load Balancer implementation in your cluster to use service type LoadBalancer. See our guide:
The Node Port method is only recommended for local clusters not exposed to the internet. The basic reason for this is insecurity of Prometheus/Alertmanager services.
There exists network policies that prevent access through Load balancer as captured in this issue. NetworkPolicies restricting access to the components are added by default. These can either be removed as in networkpolicies-disabled.jsonnet.
$ kubectl get networkpolicies
NAME POD-SELECTOR AGE
alertmanager-main app.kubernetes.io/component=alert-router,app.kubernetes.io/instance=main,app.kubernetes.io/name=alertmanager,app.kubernetes.io/part-of=kube-prometheus 35m
blackbox-exporter app.kubernetes.io/component=exporter,app.kubernetes.io/name=blackbox-exporter,app.kubernetes.io/part-of=kube-prometheus 35m
grafana app.kubernetes.io/component=grafana,app.kubernetes.io/name=grafana,app.kubernetes.io/part-of=kube-prometheus 35m
kube-state-metrics app.kubernetes.io/component=exporter,app.kubernetes.io/name=kube-state-metrics,app.kubernetes.io/part-of=kube-prometheus 35m
node-exporter app.kubernetes.io/component=exporter,app.kubernetes.io/name=node-exporter,app.kubernetes.io/part-of=kube-prometheus 35m
prometheus-adapter app.kubernetes.io/component=metrics-adapter,app.kubernetes.io/name=prometheus-adapter,app.kubernetes.io/part-of=kube-prometheus 35m
prometheus-k8s app.kubernetes.io/component=prometheus,app.kubernetes.io/instance=k8s,app.kubernetes.io/name=prometheus,app.kubernetes.io/part-of=kube-prometheus 35m
prometheus-operator app.kubernetes.io/component=controller,app.kubernetes.io/name=prometheus-operator,app.kubernetes.io/part-of=kube-prometheus 35mAn easy way out is to delete the policies even though this is not a recommended way.
$ kubectl -n monitoring delete networkpolicies.networking.k8s.io --all
networkpolicy.networking.k8s.io "alertmanager-main" deleted
networkpolicy.networking.k8s.io "blackbox-exporter" deleted
networkpolicy.networking.k8s.io "grafana" deleted
networkpolicy.networking.k8s.io "kube-state-metrics" deleted
networkpolicy.networking.k8s.io "node-exporter" deleted
networkpolicy.networking.k8s.io "prometheus-adapter" deleted
networkpolicy.networking.k8s.io "prometheus-k8s" deleted
networkpolicy.networking.k8s.io "prometheus-operator" deletedAccessing Prometheus UI and Grafana dashboard using NodePort / LoadBalancer
Prometheus:
# If you need Node Port
kubectl --namespace monitoring patch svc prometheus-k8s -p '{"spec": {"type": "NodePort"}}'
#If you have working LoadBalancer
kubectl --namespace monitoring patch svc prometheus-k8s -p '{"spec": {"type": "LoadBalancer"}}'Alertmanager:
# If you need Node Port
kubectl --namespace monitoring patch svc alertmanager-main -p '{"spec": {"type": "NodePort"}}'
#If you have working LoadBalancer
kubectl --namespace monitoring patch svc alertmanager-main -p '{"spec": {"type": "LoadBalancer"}}'Grafana:
# If you need Node Port
kubectl --namespace monitoring patch svc grafana -p '{"spec": {"type": "NodePort"}}'
#If you have working LoadBalancer
kubectl --namespace monitoring patch svc grafana -p '{"spec": {"type": "LoadBalancer"}}'Confirm that the each of the services have a Node Port assigned / Load Balancer IP addresses:
$ kubectl -n monitoring get svc | grep NodePort
alertmanager-main NodePort 10.254.220.101 <none> 9093:31237/TCP 45m
grafana NodePort 10.254.226.247 <none> 3000:31123/TCP 45m
prometheus-k8s NodePort 10.254.92.43 <none> 9090:32627/TCP 45m
$ kubectl -n monitoring get svc | grep LoadBalancer
grafana LoadBalancer 10.97.236.243 192.168.1.31 3000:30513/TCP 11mIn this example we can access the services as below:
# Grafana
NodePort: http://node_ip:31123
LB: http://lb_ip:3000
# Prometheus
NodePort: http://node_ip:31123
LB: http://lb_ip:9090
# Alert Manager
NodePort: http://node_ip:31237
LB: http://lb_ip:9093An example of default grafana dashboard showing cluster-wide compute resource usage.
Step 5: Store data in Persistent volume (Optional)
By default, the operator configures Pods to store data on emptyDir volumes which aren’t persisted when the Pods are redeployed. To maintain data across deployments and version upgrades, you can configure persistent storage for Prometheus, Alertmanager and ThanosRuler resources. For detailed how-to see customization documentation.
Install jsonnet-bundler – jsonnet package manager.
### Linux ###
VER=$(curl -s https://api.github.com/repos/jsonnet-bundler/jsonnet-bundler/releases/latest|grep tag_name|cut -d '"' -f 4|sed 's/v//')
wget https://github.com/jsonnet-bundler/jsonnet-bundler/releases/download/v${VER}/jb-linux-amd64 -O jb
chmod +x jb
sudo mv jb /usr/local/bin
### macOS ###
VER=$(curl -s https://api.github.com/repos/jsonnet-bundler/jsonnet-bundler/releases/latest|grep tag_name|cut -d '"' -f 4|sed 's/v//')
wget https://github.com/jsonnet-bundler/jsonnet-bundler/releases/download/v${VER}/jb-darwin-amd64 -O jb
chmod +x jb
sudo mv jb /usr/local/bin
#or
brew install jsonnet-bundlerDownload builder script to your local system
wget https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/main/build.sh -O build.sh
chmod +x build.shInitialize a new empty jsonnetfile
jb initInstall the kube-prometheus dependency
$ jb install github.com/prometheus-operator/kube-prometheus/jsonnet/kube-prometheus@main
GET https://github.com/prometheus-operator/kube-prometheus/archive/d3941f904503140ca2413f2ce5190381606bfb23.tar.gz 200
GET https://github.com/etcd-io/etcd/archive/04a75b10bfed11c2c093e9cf71bf4ef927f2b07c.tar.gz 200
GET https://github.com/kubernetes-monitoring/kubernetes-mixin/archive/3c386687c1f8ceb6b79ff887c4a934e9cee1b90a.tar.gz 200
GET https://github.com/kubernetes/kube-state-metrics/archive/d9677c09fb361f4d80dc2edab7c4714101cd91d8.tar.gz 200
GET https://github.com/prometheus/prometheus/archive/52adf55631c12154f3c757c9059a883382176945.tar.gz 200
GET https://github.com/pyrra-dev/pyrra/archive/e3d0e76e16bd83c3ca886d4e44e21ff590d054a8.tar.gz 200
GET https://github.com/thanos-io/thanos/archive/a89dd0c276ab9a9d7423aa3688d3f3de082bc968.tar.gz 200
GET https://github.com/prometheus/alertmanager/archive/8c28ce521525fd7f0a7b1759bbcb6712583a5c0a.tar.gz 200
GET https://github.com/brancz/kubernetes-grafana/archive/d039275e4916aceae1c137120882e01d857787ac.tar.gz 200
GET https://github.com/grafana/grafana/archive/1120f9e255760a3c104b57871fcb91801e934382.tar.gz 200
GET https://github.com/prometheus-operator/prometheus-operator/archive/03ab344439bf30c2686f498b5a5a5b7e74840f84.tar.gz 200
GET https://github.com/prometheus-operator/prometheus-operator/archive/03ab344439bf30c2686f498b5a5a5b7e74840f84.tar.gz 200
GET https://github.com/kubernetes/kube-state-metrics/archive/d9677c09fb361f4d80dc2edab7c4714101cd91d8.tar.gz 200
GET https://github.com/prometheus/node_exporter/archive/22e5dd2f0d088208d7980b43e17a984104c0e6a6.tar.gz 200
GET https://github.com/grafana/jsonnet-libs/archive/d68f9a6e0b1af7c4c4056dc2b43fb8f3bac01f43.tar.gz 200
GET https://github.com/grafana/grafonnet-lib/archive/30280196507e0fe6fa978a3e0eaca3a62844f817.tar.gz 200
GET https://github.com/grafana/grafonnet-lib/archive/30280196507e0fe6fa978a3e0eaca3a62844f817.tar.gz 200Install go-jsonnet dependency
### Linux ###
VER=$(curl -s https://api.github.com/repos/google/go-jsonnet/releases/latest|grep tag_name|cut -d '"' -f 4|sed 's/v//')
wget https://github.com/google/go-jsonnet/releases/download/v${VER}/go-jsonnet_${VER}_Linux_x86_64.tar.gz
tar xvf go-jsonnet_${VER}_Linux_x86_64.tar.gz
sudo mv jsonnet jsonnetfmt /usr/local/bin
### macOS ###
VER=$(curl -s https://api.github.com/repos/google/go-jsonnet/releases/latest|grep tag_name|cut -d '"' -f 4|sed 's/v//')
wget https://github.com/google/go-jsonnet/releases/download/v${VER}/go-jsonnet_${VER}_Darwin_x86_64.tar.gz
tar xvf go-jsonnet_${VER}_Darwin_x86_64.tar.gz
sudo mv jsonnet jsonnetfmt /usr/local/binAlso install gojsontoyaml dependency.
VER=$(curl -s https://api.github.com/repos/brancz/gojsontoyaml/releases/latest|grep tag_name|cut -d '"' -f 4|sed 's/v//')
wget https://github.com/brancz/gojsontoyaml/releases/download/v${VER}/gojsontoyaml_${VER}_darwin_amd64.tar.gz
wget https://github.com/brancz/gojsontoyaml/releases/download/v${VER}/gojsontoyaml_${VER}_linux_amd64.tar.gz
tar xvf gojsontoyaml_${VER}_darwin_amd64.tar.gz
sudo mv gojsontoyaml /usr/local/binCreate PVC jsonnet file.
$ vim prometheus-pvc.jsonnet
local kp =
(import 'kube-prometheus/main.libsonnet') +
// Uncomment the following imports to enable its patches
// (import 'kube-prometheus/addons/anti-affinity.libsonnet') +
// (import 'kube-prometheus/addons/managed-cluster.libsonnet') +
// (import 'kube-prometheus/addons/node-ports.libsonnet') +
// (import 'kube-prometheus/addons/static-etcd.libsonnet') +
{
values+:: {
common+: {
namespace: 'monitoring',
},
},
prometheus+:: {
prometheus+: {
spec+: { // https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#prometheusspec
// If a value isn't specified for 'retention', then by default the '--storage.tsdb.retention=24h' arg will be passed to prometheus by prometheus-operator.
// The possible values for a prometheus <duration> are:
// * https://github.com/prometheus/common/blob/c7de230/model/time.go#L178 specifies "^([0-9]+)(y|w|d|h|m|s|ms)$" (years weeks days hours minutes seconds milliseconds)
retention: '30d',
// Reference info: https://github.com/coreos/prometheus-operator/blob/master/Documentation/user-guides/storage.md
// By default (if the following 'storage.volumeClaimTemplate' isn't created), prometheus will be created with an EmptyDir for the 'prometheus-k8s-db' volume (for the prom tsdb).
// This 'storage.volumeClaimTemplate' causes the following to be automatically created (via dynamic provisioning) for each prometheus pod:
// * PersistentVolumeClaim (and a corresponding PersistentVolume)
// * the actual volume (per the StorageClassName specified below)
storage: { // https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#storagespec
volumeClaimTemplate: { // https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.11/#persistentvolumeclaim-v1-core (defines variable named 'spec' of type 'PersistentVolumeClaimSpec')
apiVersion: 'v1',
kind: 'PersistentVolumeClaim',
spec: {
accessModes: ['ReadWriteOnce'],
// https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.11/#resourcerequirements-v1-core (defines 'requests'),
// and https://kubernetes.io/docs/concepts/policy/resource-quotas/#storage-resource-quota (defines 'requests.storage')
resources: { requests: { storage: '100Gi' } },
// A StorageClass of the following name (which can be seen via `kubectl get storageclass` from a node in the given K8s cluster) must exist prior to kube-prometheus being deployed.
storageClassName: 'nfs-01',
// The following 'selector' is only needed if you're using manual storage provisioning (https://github.com/coreos/prometheus-operator/blob/master/Documentation/user-guides/storage.md#manual-storage-provisioning).
// And note that this is not supported/allowed by AWS - uncommenting the following 'selector' line (when deploying kube-prometheus to a K8s cluster in AWS) will cause the pvc to be stuck in the Pending status and have the following error:
// * 'Failed to provision volume with StorageClass "ssd": claim.Spec.Selector is not supported for dynamic provisioning on AWS'
// selector: { matchLabels: {} },
},
},
}, // storage
}, // spec
}, // prometheus
}, // prometheus
};
{ ['setup/0namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{
['setup/prometheus-operator-' + name]: kp.prometheusOperator[name]
for name in std.filter((function(name) name != 'serviceMonitor'), std.objectFields(kp.prometheusOperator))
} +
// serviceMonitor is separated so that it can be created after the CRDs are ready
{ 'prometheus-operator-serviceMonitor': kp.prometheusOperator.serviceMonitor } +
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['prometheus-adapter-' + name]: kp.prometheusAdapter[name] for name in std.objectFields(kp.prometheusAdapter) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) }Substitute:
- 30d with your desired retention period.
- 100Gi with the size of PVC to provision
- nfs-01 with StorageClass name
Now compile the manifests
$ ./build.sh prometheus-pvc.jsonnet
+ set -o pipefail
++ pwd
+ PATH='/Users/jkmutai/myhacks/kube-prometheus-customizations/tmp/bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/VMware Fusion.app/Contents/Public'
+ rm -rf manifests
+ mkdir -p manifests/setup
+ jsonnet -J vendor -m manifests prometheus-pvc.jsonnet
+ xargs '-I{}' sh -c 'cat {} | gojsontoyaml > {}.yaml' -- '{}'
+ find manifests -type f '!' -name '*.yaml' -delete
+ rm -f kustomizationRun the following command to apply resource customization.
kubectl apply -f manifests/prometheus-prometheus.yamlThen proceed to create the remaining resources
kubectl apply -f manifests/Confirm PVCs were created and now being used to store data.
$ kubectl get pvc -n monitoring
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
prometheus-k8s-db-prometheus-k8s-0 Bound pvc-6e270bdb-e815-4e9e-b7bf-b5266ba651f1 100Gi RWO nfs-01 7m17s
prometheus-k8s-db-prometheus-k8s-1 Bound pvc-3f38c9c5-81c6-4d7f-b71b-4e980e7152b5 100Gi RWO nfs-01 7m17sContents added to original yaml are as shown below.
$ vim manifests/prometheus-prometheus.yaml
...
spec:
storage:
volumeClaimTemplate:
apiVersion: v1
kind: PersistentVolumeClaim
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100Gi
storageClassName: nfs-01Destroying / Tearing down Prometheus monitoring stack
If at some point you feel like tearing down Prometheus Monitoring stack in your Kubernetes Cluster, you can run kubectl delete command and pass the path to the manifest files we used during deployment.
kubectl delete --ignore-not-found=true -f manifests/ -f manifests/setupWithin some few minutes the stack is deleted and you can re-deploy if that was the intention.
More guides on Kubernetes / Containers:
- Install Ambassador Ingress Controller on Kubernetes
- Deploy and Use OpenEBS Container Storage on Kubernetes
- Authenticate Kubernetes Dashboard Users With Active Directory
- Automatically replace unhealthy nodes on Magnum Kubernetes using magnum-auto-healer
- Create Kubernetes Cluster on OpenStack Magnum with Fedora CoreOS
Leave a Reply
You must be logged in to post a comment.