Vendor the base# Run these commands from the root of your Kubestack infra repositorywget https://storage.googleapis.com/catalog.kubestack.com/prometheus-v0.41.1-kbst.0.zipunzip -d manifests/bases/ prometheus-v0.41.1-kbst.0.ziprm prometheus-v0.41.1-kbst.0.zip
Include resource in apps overlaycd manifests/overlays/appskustomize edit add resource ../../bases/prometheus/clusterwide
Commit and pushcd -git checkout -b add-prometheusgit add manifests/bases/prometheus manifests/overlays/apps/kustomization.yamlgit commit -m "Add prometheus v0.41.1-kbst.0 base"git push origin add-prometheus
Review PR and merge
Finally, review and merge the PR into master. Once it's been successfully applied against the Ops-Cluster set a prod-deploy tag to also apply the change against the Apps-Cluster.
To update the operator delete the previously vendored base and then vendor the new version.
Delete the previous vendored version# Run these commands from the root of your Kubestack infra repositoryrm -r manifests/bases/prometheus
Vendor the new version# Run these commands from the root of your Kubestack infra repositorywget https://storage.googleapis.com/catalog.kubestack.com/prometheus-v0.41.1-kbst.0.zipunzip -d manifests/bases/ prometheus-v0.41.1-kbst.0.ziprm prometheus-v0.41.1-kbst.0.zip
Commit and pushgit checkout -b update-prometheusgit add manifests/bases/prometheusgit commit -m "Update prometheus base to v0.41.1-kbst.0"git push origin update-prometheus
Operators often create resources based on custom objects. When removing an operator, follow a two-step process to ensure operator provisioned resources are purged properly.
- Remove all the operator's custom objects.
- Once the operator had time to de-provision all resources it created, follow the instructions below to remove the operator itself.
Remove resource from apps overlaycd manifests/overlays/appskustomize edit remove resource ../../bases/prometheus/clusterwide
Delete the vendored base from your repositorycd -# Run these commands from the root of your Kubestack infra repositoryrm -r manifests/bases/prometheus
Commit and pushgit checkout -b remove-prometheusgit add manifests/bases/prometheusgit commit -m "Remove prometheus base"git push origin remove-prometheus
Using the Prometheus operator to provision a Prometheus instance and start monitoring an application requires three steps:
- Deploy the operator
- Provision a Prometheus instance
- Create ServiceMonitors for each service exposing metrics
The first step was to deploy the Prometheus operator following the instructions on the install tab.
Prometheus Instance Manifests
Next, we can continue with step number two and use the operator to provision our Prometheus instance.
Each Prometheus instance needs read-only access to the Kubernetes api in order to keep its monitoring targets up to date. To make this easier, the clusterwide base includes two
ClusterRoles. One for the operator, called
prometheus-operator, and one for the instances, aptly named
So, in order to provision a Prometheus instance we need to first create a
ServiceAccount and a
RoleBinding linking that service account to the
pormetheus-instance cluster role. Finally, we can create a
Prometheus resource that instructs the operator to provision a Prometheus instance that uses our service account. Below example does just that.
---apiVersion: monitoring.coreos.com/v1kind: Prometheusmetadata:name: example-instancenamespace: defaultlabels:prometheus: example-instancespec:serviceAccountName: prometheus-example-instanceserviceMonitorSelector:matchLabels:prometheus-instance: example-instanceresources:requests:# by default the operator requests 2Gi of memory# adapt the line below if required to schedule podsmemory: 2Gi---apiVersion: rbac.authorization.k8s.io/v1beta1kind: RoleBindingmetadata:name: prometheus-example-instancenamespace: defaultroleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: prometheus-instancesubjects:- kind: ServiceAccountname: prometheus-example-instancenamespace: default---apiVersion: v1kind: ServiceAccountmetadata:name: prometheus-example-instancenamespace: default
Finally, we need to create a
ServiceMonitor to instruct the operator, to configure our Prometheus instance to scrape metrics from each of the replicas of an application that exposes its metrics in the Prometheus format.
Since this step is depending on the application and where it exposes the metrics, you will have to adapt the example below.
But there are basically three things to note. First, the
metadata.labels need to match the
spec.serviceMonitorSelector.matchLabels from the
Prometheus resource. Next, the
spec.selector.matchLabels below needs to match the
metadata.labels set for the service of the application you are trying to monitor. Finally, the
spec.endpoints need to match the port name, configured in the application's service.
apiVersion: monitoring.coreos.com/v1kind: ServiceMonitormetadata:name: application-examplelabels:# this label instructs Prometheus to include this ServiceMonitor# based on the `spec.serviceMonitorSelector.matchLabels` aboveprometheus-instance: example-instancespec:selector:matchLabels:# this selector is how the `ServiceMonitor` finds the# application's serviceapp: application-exampleendpoints:# this tells Prometheus on what port the app exposes its metrics- port: web
To check if Promethues has started scraping your applicaiton's metrics, you can access the Prometheus UI using
kubectl port-forward and use it to run a query.
kubectl port-forward prometheus-example-instance-0 9090
Above instructions only cover the basics. The Prometheus operator has many more features and options. For more extended documentation about Prometheus operator configuration, please refer to the official documentation.