Prometheus Operator
Terraform module for Kubernetes

The Prometheus operator provisions and configures the popular Prometheus monitoring system and time series database on top of Kubernetes. It allows setting both Prometheus configuration like versions, persistence, retention policies as well as monitoring targets through Kubernetes custom resources and manages the lifecycle of the Prometheus instances.

You can use the operator to provision Prometheus instances to monitor any workload that exposes metrics in the Prometheus format. Additionally the operator can also provision AlertManager for alerting.

This Terraform module provisions Prometheus Operator on Kubernetes. It fully integrates the Kubernetes resources into the Terraform workflow and allows configuring Prometheus Operator using native Terraform syntax.

The Prometheus Operator module is continuously updated and tested when new upstream versions are released.

TL;DR:

  • Call the module once per desired target cluster.
  • The provider alias you pass into the module determines the cluster.
  • Customize the Kubernetes resources per environment using Terraform syntax.

Use the module

To provision the Prometheus Operator Terraform module on a Kubernetes cluster, call the module, set source and version, and pass an aliased kustomization provider into the module. The provider configuration determines what cluster the Kuberneters resources are provisioned on. Framework documentation includes a complete example of how the kubeconfig output of a cluster module can be used to configure a kustomization provider alias.

module "eks_zero_prometheus" {
providers = {
kustomization = kustomization.eks_zero
}
source = "kbst.xyz/catalog/prometheus/kustomization"
version = "0.51.2-kbst.0"
}
module "aks_zero_prometheus" {
providers = {
kustomization = kustomization.aks_zero
}
source = "kbst.xyz/catalog/prometheus/kustomization"
version = "0.51.2-kbst.0"
}
module "gke_zero_prometheus" {
providers = {
kustomization = kustomization.gke_zero
}
source = "kbst.xyz/catalog/prometheus/kustomization"
version = "0.51.2-kbst.0"
}

Customize resources

All Kubestack cluster service modules support the same module attributes and per environment configuration. The module configuration is a Kustomization set in the per environment configuration map following Kubestack's inheritance model.

This example overwrites the metadata.namespace of all Kubernetes resources provisioned by the Prometheus Operator module using a Terraform variable.

module "example_prometheus" {
providers = {
kustomization = kustomization.example
}
source = "kbst.xyz/catalog/prometheus/kustomization"
version = "0.51.2-kbst.0"
configuration = {
apps = {
namespace = var.example_prometheus_namespace
}
ops = {}
loc = {}
}
}

Full documentation how to customize a module's Kubernetes resources is available in the cluster service module configuration section of the framework documentation.

Usage

Using the Prometheus operator to provision a Prometheus instance and start monitoring an application requires three steps:

  1. Deploy the operator
  2. Provision a Prometheus instance
  3. Create ServiceMonitors for each service exposing metrics

The first step was to deploy the Prometheus operator following the instructions on the install tab.

Prometheus Instance Manifests

Next, we can continue with step number two and use the operator to provision our Prometheus instance.

Each Prometheus instance needs read-only access to the Kubernetes api in order to keep its monitoring targets up to date. To make this easier, the clusterwide base includes two ClusterRoles. One for the operator, called prometheus-operator, and one for the instances, aptly named prometheus-instance.

So, in order to provision a Prometheus instance we need to first create a ServiceAccount and a RoleBinding linking that service account to the pormetheus-instance cluster role. Finally, we can create a Prometheus resource that instructs the operator to provision a Prometheus instance that uses our service account. Below example does just that.

---
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
name: example-instance
namespace: default
labels:
prometheus: example-instance
spec:
serviceAccountName: prometheus-example-instance
serviceMonitorSelector:
matchLabels:
prometheus-instance: example-instance
resources:
requests:
# by default the operator requests 2Gi of memory
# adapt the line below if required to schedule pods
memory: 2Gi
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: prometheus-example-instance
namespace: default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: prometheus-instance
subjects:
- kind: ServiceAccount
name: prometheus-example-instance
namespace: default
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: prometheus-example-instance
namespace: default

ServiceMonitors Manifest

Finally, we need to create a ServiceMonitor to instruct the operator, to configure our Prometheus instance to scrape metrics from each of the replicas of an application that exposes its metrics in the Prometheus format.

Since this step is depending on the application and where it exposes the metrics, you will have to adapt the example below.

But there are basically three things to note. First, the metadata.labels need to match the spec.serviceMonitorSelector.matchLabels from the Prometheus resource. Next, the spec.selector.matchLabels below needs to match the metadata.labels set for the service of the application you are trying to monitor. Finally, the spec.endpoints need to match the port name, configured in the application's service.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: application-example
labels:
# this label instructs Prometheus to include this ServiceMonitor
# based on the `spec.serviceMonitorSelector.matchLabels` above
prometheus-instance: example-instance
spec:
selector:
matchLabels:
# this selector is how the `ServiceMonitor` finds the
# application's service
app: application-example
endpoints:
# this tells Prometheus on what port the app exposes its metrics
- port: web

Prometheus UI

To check if Promethues has started scraping your applicaiton's metrics, you can access the Prometheus UI using kubectl port-forward and use it to run a query.

kubectl port-forward prometheus-example-instance-0 9090

Advanced Configuration

Above instructions only cover the basics. The Prometheus operator has many more features and options. For more extended documentation about Prometheus operator configuration, please refer to the official documentation.