Prometheus rules, alerts, and Grafana dashboards

[P4sca1]

https://github.com/kubernetes-monitoring/kubernetes-mixin provides a set of Grafana dashboards and Prometheus alerts for Kubernetes.
It works well in tandem with this helm chart. I successfully deployed dashboards and alerting rules and want to share with the community how it is done.

Maybe instructions can be added to the README or even better maybe this could be added to the helm chart via a feature flag.

Prerequisites

Install jsonnet and jsonnet-bundler.

# On MacOS using brew
brew install jsonnet jsonnet-bundler

Install dependencies

# This will produce a jsonnetfile.json file in the current working directory
jb init

# This will install the kubernetes-mixin to the ./vendor directory
jb install https://github.com/kubernetes-monitoring/kubernetes-mixin

# make sure to add the vendor directory to your .gitignore file

Add the mixin.libsonnet file

The mixin.libsonnet file contains the jsonnet configuration. The selectors need to be updated to use the labels that are configured by this helm chart for Grafana Alloy. The below configuration is compatible with the default values used by this helm chart.

local kubernetes = import "kubernetes-mixin/mixin.libsonnet";

kubernetes {
  _config+:: {
    cadvisorSelector: 'job="integrations/kubernetes/cadvisor"',
    kubeletSelector: 'job="integrations/kubernetes/kubelet"',
    kubeStateMetricsSelector: 'job="integrations/kubernetes/kube-state-metrics"',
    nodeExporterSelector: 'job="integrations/node_exporter"',
    kubeSchedulerSelector: 'job="kube-scheduler"',
    kubeControllerManagerSelector: 'job="kube-controller-manager"',
    kubeApiserverSelector: 'job="integrations/kubernetes/kube-apiserver"',
    kubeProxySelector: 'job="integrations/kubernetes/kube-proxy"',
    podLabel: 'pod',
    hostNetworkInterfaceSelector: 'device!~"veth.+"',
    hostMountpointSelector: 'mountpoint="/"',
    windowsExporterSelector: 'job="integrations/windows_exporter"',
    containerfsSelector: 'container!=""',

    grafanaK8s+:: {
      dashboardNamePrefix: '',
      dashboardTags: ['kubernetes', 'infrastructure'],
    },
  },
}

Generate alerts, rules, and dashboards

Run the following commands to generate the files.

mkdir -p files/dashboards
jsonnet -J vendor -S -e 'std.manifestYamlDoc((import "mixin.libsonnet").prometheusAlerts)' > files/alerts.yml
jsonnet -J vendor -S -e 'std.manifestYamlDoc((import "mixin.libsonnet").prometheusRules)' > files/rules.yml
jsonnet -J vendor -m files/dashboards -e '(import "mixin.libsonnet").grafanaDashboards'

You then need to apply the generated files to prometheus / grafana. I am personally using the Prometheus and Grafana operator for that purpose.

Add operator CRDs using helm

# Prometheus rules
---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: kubernetes-mixin-rules
spec:
{{- .Files.Get "files/rules.yml" | fromYaml | toYaml | nindent 2 }}

# Prometheus alerts
---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: kubernetes-mixin-alerts
spec:
{{- .Files.Get "files/alerts.yml" | fromYaml | toYaml | nindent 2 }}

# Grafana dashboards
{{- range $path, $_ :=  .Files.Glob  "files/dashboards/*.json" -}}
{{- $baseName := $path | base | trimSuffix (ext $path) }}
---
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  name: "kubernetes-mixin-{{ $baseName }}-dashboard"
spec:
  instanceSelector:
    matchLabels:
      grafana: main
  json: |-
    {{- $.Files.Get $path | nindent 4 }}
{{- end }}

[dgamo]

Great job! I have been using that mixin for many years, and it's great to have such an easy way to integrate it with Grafana Cloud