googleforgames / googleforgames/open-match

Add Prometheus Operator ServiceMonitor objects to the Helm chart

Open
#1,245 1 comment 0 reactions 0 assignees View on GitHub
area/developer-experience area/feature
Dominant language
Go
Stars
3.4k
Forks
359
PR merge metrics
No merged PRs in 30d

Description

**Is your feature request related to a problem? Please describe.**

We run Prometheus Operator, without deployment-specific Prometheus customisations, but we'd like to collect metrics from OpenMatch deployments installed with Helm.

**Describe the solution you'd like**

When using a Helm chart, being able to enable a `ServiceMonitor` via a values.yaml e.g. `global.tememetry.prometheus.serviceMonitors`, for each `Service` that selects `Pods` from a `Deployment` that uses the `prometheus.annotations` template currently.

**Describe alternatives you've considered**

Customising every deployment of Prometheus on each of our clusters to scrape any Pod with `prometheus.io/scrape` annotation. As well as requiring customisation on every cluster, this is slower as annotations are not indexed, and so every Pod on the cluster must be scanned, while `ServiceMonitors` select endpoints by Service labels, which are indexed for exactly this use-case.

The existing Helm chart includes a [customised Prometheus deployment](https://github.com/googleforgames/open-match/blob/3334f7f74a2670ddead001d8678a2991f929cb68/install/helm/open-match/subcharts/open-match-telemetry/values.yaml#L34-L126), so we could deploy that and customise our global Prometheus to federate from that. This still requires custom Prometheus configuration. It also means we're running _another_ Prometheus stack inside every open-match stack, and the current config supplied will scrape any Pods on the cluster with those annotations, not just the Open Match-owned Pods, and also scrapes the k8s API server.

**Additional context**

There has been some historical confusion, but the annotation-based scheme is not recommended by Prometheus developers ([a](https://github.com/coreos/prometheus-operator/issues/38#issuecomment-389922989) [b](https://github.com/coreos/prometheus-operator/issues/1547#issuecomment-401092041)). The specific `prometheus.io/scrape` annotation used here is [an elevated example](https://github.com/googleforgames/open-match/blob/master/install/helm/open-match/subcharts/open-match-telemetry/values.yaml#L93). It's not supposed to be the globally-used annotation for pod-scraping, as users should not be using arbitrary annotations in the `prometheus.io` namespace. I have seen a reference for the latter statement from Prometheus developers, but can't lay my hands on it right now.

Edit: Found the reference for "`prometheus.io/scrape` annotation is just an example": `prometheus.io/scrape` was [removed from the Prometheus examples in 2018](https://github.com/prometheus/prometheus/commit/03a9e7f72e072c6d29f422425d8acd91a957836b) to clarify this, replaced with `example.io/should_be_scraped`.

Contributor guide

Open the contributing guide

Research direction

Start with install/helm/open-match/subcharts/open-match-telemetry/values.yaml and the surrounding Helm chart templates, especially the existing Prometheus annotations and Services. Determine how a values.yaml setting can enable ServiceMonitor objects for the relevant Services; done means Helm installs the configurable ServiceMonitors so Prometheus Operator can discover Open Match endpoints without the annotation-based scrape configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
helm, kubernetes, prometheus
Domain
devops, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.