GoogleCloudPlatform / GoogleCloudPlatform/prometheus-engine
[bug] Unable to specify endpoints with the same port name
- Dominant language
- Go
- Stars
- 232
- Forks
- 109
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 27
Description
I convert a ServiceMonitor resource rendered from a helm chart into a PodMonitoring resource according to the [GMP documentation](https://cloud.google.com/stackdriver/docs/managed-prometheus/setup-managed#gmp-servicemonitor).
The initial ServceMonitor resource
```yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
# Removed
#annotations:
labels:
app.kubernetes.io/component: keycloak
app.kubernetes.io/instance: keycloak
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: keycloak
helm.sh/chart: keycloak-15.1.1
name: keycloak
namespace: keycloak
spec:
endpoints:
- interval: 30s
path: /metrics
port: http
- interval: 30s
path: /realms/master/metrics
port: http
namespaceSelector:
matchNames:
- keycloak
selector:
matchLabels:
app.kubernetes.io/component: metrics
app.kubernetes.io/instance: keycloak
app.kubernetes.io/name: keycloak
```
The resulting PodMonitoring resource
```yaml
apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
# Removed
#annotations:
labels:
app.kubernetes.io/component: keycloak
app.kubernetes.io/instance: keycloak
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: keycloak
helm.sh/chart: keycloak-15.1.1
name: keycloak
namespace: keycloak
spec:
endpoints:
- interval: 30s
path: /metrics
port: http
- interval: 30s
path: /realms/master/metrics
port: http
selector:
matchLabels:
app.kubernetes.io/instance: keycloak
app.kubernetes.io/name: keycloak
```
The difference
```diff
--- servicemonitor-keycloak.yaml 2023-05-19 13:43:45.708860839 +0200
+++ podmonitoring-keycloak.yaml 2023-05-19 13:47:15.243825855 +0200
@@ -1,31 +1,27 @@
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
+apiVersion: monitoring.googleapis.com/v1
+kind: PodMonitoring
metadata:
# Removed
#annotations:
labels:
app.kubernetes.io/component: keycloak
app.kubernetes.io/instance: keycloak
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: keycloak
helm.sh/chart: keycloak-15.1.1
name: keycloak
namespace: keycloak
spec:
endpoints:
- interval: 30s
path: /metrics
port: http
- interval: 30s
path: /realms/master/metrics
port: http
- namespaceSelector:
- matchNames:
- - keycloak
selector:
matchLabels:
- app.kubernetes.io/component: metrics
app.kubernetes.io/instance: keycloak
app.kubernetes.io/name: keycloak
```
After applying the PodMonitoring resource to my environment I see the following errors in collector pods:
```
prometheus ts=2023-05-19T11:36:08.511Z caller=main.go:1198 level=info msg="Loading configuration file" filename=/prometheus/config_out/config.yaml
prometheus ts=2023-05-19T11:36:08.514Z caller=main.go:929 level=error msg="Error reloading config" err="couldn't load configuration (--config.file=\"/prometheus/config_out/config.yaml\"): parsing YAML file /prometheus/config_out/config.yaml: found multiple scrape configs with job name \"PodMonitoring/keycloak/keycloak/http\""
```
```
config-reloader level=error ts=2023-05-19T11:36:18.51061949Z caller=runutil.go:101 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post \"http://localhost:19090/-/reload\": context deadline exceeded"
config-reloader level=error ts=2023-05-19T11:36:18.514124586Z caller=reloader.go:382 msg="Failed to trigger reload. Retrying." err="trigger reload: reload request failed: Post \"http://localhost:19090/-/reload\": context deadline exceeded"
config-reloader level=error ts=2023-05-19T11:36:18.517247779Z caller=runutil.go:101 msg="function failed. Retrying in next tick" err="trigger reload: received non-200 response: 500 Internal Server Error; have you set `--web.enable-lifecycle` Prometheus flag?"
```
The most interesting here seems to be this error:
```
found multiple scrape configs with job name "PodMonitoring/keycloak/keycloak/http"
```
I assume that the logic of generating job names doesn't consider endpoint paths, and there must be no endpoints with the same port name. Is that correct?
As a workaround, I'll create one PodMonitoring resource per port.
Contributor guide
Assessment
This issue has not been assessed yet.