Support and expose an SLO-based monitoring interface

(Late comment on the proposal: https://github.com/openshift/enhancements/pull/866#pullrequestreview-771897761)

Hello, I am part of the MTSRE team at Red Hat and we would like to expose an SLO-based monitoring interface to our tenants. The goals behind this request are:

• Be able to easily measure SLOs/SLIs and error budgets
• Abstract as much of the monitoring stack as possible to our MTO (prometheus/alertmanager/grafana configs) so that they can spend their time doing feature development and bug fixing
• Make sure we perform active reconciliation on this interface (operator-pattern)

There exists various tools out there that could help us abstract away prometheus/alertmanager and grafana configs:

Sloth: https://github.com/slok/sloth
Keptn: https://github.com/keptn/keptn

I personally like sloth a lot, as it already has a kubernetes operator and an SLO-based CRD: PrometheusServiceLevel. Also, it maintains a set of uniform grafana dashboards, that can be automatically deployed: https://sloth.dev/dashboards/.

Benefits of an SLO-based monitoring approach:

• Provide better context on what you monitor, and put your users first
• Abstract away the monitoring stack, as prometheus/alertmanager/grafana configs are challenging
• Provide a uniform visualization interface by reusing the same grafana dashboards for all addons. This is a big plus for me as developing dashboards is extremely time-consuming, MTSRE engineers will get a huge productivity boost from an homogeneous visualization interface.
• Reconcile SLOs and generated monitoring configs, to allow evolution and iteration (operator-pattern)

This is also the approach that was chosen by the AppSRE team, see /schemas/app-sre/slo-document-1.yml in app-interface.

Start by reading the linked observability-enhancements proposal and the referenced Sloth Kubernetes operator and PrometheusServiceLevel examples. Review /schemas/app-sre/slo-document-1.yml in app-interface for the mentioned SLO approach. Done would require an agreed SLO-based interface with active reconciliation and the promised monitoring-stack abstraction.