googleforgames / googleforgames/open-match
Add Prometheus Operator ServiceMonitor objects to the Helm chart
- Dominant language
- Go
- Stars
- 3.4k
- Forks
- 359
- PR merge metrics
- No merged PRs in 30d
Description
**Is your feature request related to a problem? Please describe.**
We run Prometheus Operator, without deployment-specific Prometheus customisations, but we'd like to collect metrics from OpenMatch deployments installed with Helm.
**Describe the solution you'd like**
When using a Helm chart, being able to enable a `ServiceMonitor` via a values.yaml e.g. `global.tememetry.prometheus.serviceMonitors`, for each `Service` that selects `Pods` from a `Deployment` that uses the `prometheus.annotations` template currently.
**Describe alternatives you've considered**
Customising every deployment of Prometheus on each of our clusters to scrape any Pod with `prometheus.io/scrape` annotation. As well as requiring customisation on every cluster, this is slower as annotations are not indexed, and so every Pod on the cluster must be scanned, while `ServiceMonitors` select endpoints by Service labels, which are indexed for exactly this use-case.
The existing Helm chart includes a [customised Prometheus deployment](https://github.com/googleforgames/open-match/blob/3334f7f74a2670ddead001d8678a2991f929cb68/install/helm/open-match/subcharts/open-match-telemetry/values.yaml#L34-L126), so we could deploy that and customise our global Prometheus to federate from that. This still requires custom Prometheus configuration. It also means we're running _another_ Prometheus stack inside every open-match stack, and the current config supplied will scrape any Pods on the cluster with those annotations, not just the Open Match-owned Pods, and also scrapes the k8s API server.
**Additional context**
There has been some historical confusion, but the annotation-based scheme is not recommended by Prometheus developers ([a](https://github.com/coreos/prometheus-operator/issues/38#issuecomment-389922989) [b](https://github.com/coreos/prometheus-operator/issues/1547#issuecomment-401092041)). The specific `prometheus.io/scrape` annotation used here is [an elevated example](https://github.com/googleforgames/open-match/blob/master/install/helm/open-match/subcharts/open-match-telemetry/values.yaml#L93). It's not supposed to be the globally-used annotation for pod-scraping, as users should not be using arbitrary annotations in the `prometheus.io` namespace. I have seen a reference for the latter statement from Prometheus developers, but can't lay my hands on it right now.
Edit: Found the reference for "`prometheus.io/scrape` annotation is just an example": `prometheus.io/scrape` was [removed from the Prometheus examples in 2018](https://github.com/prometheus/prometheus/commit/03a9e7f72e072c6d29f422425d8acd91a957836b) to clarify this, replaced with `example.io/should_be_scraped`.
Contributor guide
Research direction
Start with install/helm/open-match/subcharts/open-match-telemetry/values.yaml and the surrounding Helm chart templates, especially the existing Prometheus annotations and Services. Determine how a values.yaml setting can enable ServiceMonitor objects for the relevant Services; done means Helm installs the configurable ServiceMonitors so Prometheus Operator can discover Open Match endpoints without the annotation-based scrape configuration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- helm, kubernetes, prometheus
- Domain
- devops, observability
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100