GoogleCloudPlatform / GoogleCloudPlatform/prometheus-engine

[bug] Unable to specify endpoints with the same port name

Open
#479 13 comments 3 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
232
Forks
109
Avg merge
1d 3h
Merged PRs (30d)
27

Description

I convert a ServiceMonitor resource rendered from a helm chart into a PodMonitoring resource according to the [GMP documentation](https://cloud.google.com/stackdriver/docs/managed-prometheus/setup-managed#gmp-servicemonitor).

The initial ServceMonitor resource

```yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
# Removed
#annotations:
labels:
app.kubernetes.io/component: keycloak
app.kubernetes.io/instance: keycloak
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: keycloak
helm.sh/chart: keycloak-15.1.1
name: keycloak
namespace: keycloak
spec:
endpoints:
- interval: 30s
path: /metrics
port: http
- interval: 30s
path: /realms/master/metrics
port: http
namespaceSelector:
matchNames:
- keycloak
selector:
matchLabels:
app.kubernetes.io/component: metrics
app.kubernetes.io/instance: keycloak
app.kubernetes.io/name: keycloak
```


The resulting PodMonitoring resource

```yaml
apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
# Removed
#annotations:
labels:
app.kubernetes.io/component: keycloak
app.kubernetes.io/instance: keycloak
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: keycloak
helm.sh/chart: keycloak-15.1.1
name: keycloak
namespace: keycloak
spec:
endpoints:
- interval: 30s
path: /metrics
port: http
- interval: 30s
path: /realms/master/metrics
port: http
selector:
matchLabels:
app.kubernetes.io/instance: keycloak
app.kubernetes.io/name: keycloak
```

The difference

```diff
--- servicemonitor-keycloak.yaml 2023-05-19 13:43:45.708860839 +0200
+++ podmonitoring-keycloak.yaml 2023-05-19 13:47:15.243825855 +0200
@@ -1,31 +1,27 @@
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
+apiVersion: monitoring.googleapis.com/v1
+kind: PodMonitoring
metadata:
# Removed
#annotations:
labels:
app.kubernetes.io/component: keycloak
app.kubernetes.io/instance: keycloak
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: keycloak
helm.sh/chart: keycloak-15.1.1
name: keycloak
namespace: keycloak
spec:
endpoints:
- interval: 30s
path: /metrics
port: http
- interval: 30s
path: /realms/master/metrics
port: http
- namespaceSelector:
- matchNames:
- - keycloak
selector:
matchLabels:
- app.kubernetes.io/component: metrics
app.kubernetes.io/instance: keycloak
app.kubernetes.io/name: keycloak
```

After applying the PodMonitoring resource to my environment I see the following errors in collector pods:

```
prometheus ts=2023-05-19T11:36:08.511Z caller=main.go:1198 level=info msg="Loading configuration file" filename=/prometheus/config_out/config.yaml
prometheus ts=2023-05-19T11:36:08.514Z caller=main.go:929 level=error msg="Error reloading config" err="couldn't load configuration (--config.file=\"/prometheus/config_out/config.yaml\"): parsing YAML file /prometheus/config_out/config.yaml: found multiple scrape configs with job name \"PodMonitoring/keycloak/keycloak/http\""
```

```
config-reloader level=error ts=2023-05-19T11:36:18.51061949Z caller=runutil.go:101 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post \"http://localhost:19090/-/reload\": context deadline exceeded"
config-reloader level=error ts=2023-05-19T11:36:18.514124586Z caller=reloader.go:382 msg="Failed to trigger reload. Retrying." err="trigger reload: reload request failed: Post \"http://localhost:19090/-/reload\": context deadline exceeded"
config-reloader level=error ts=2023-05-19T11:36:18.517247779Z caller=runutil.go:101 msg="function failed. Retrying in next tick" err="trigger reload: received non-200 response: 500 Internal Server Error; have you set `--web.enable-lifecycle` Prometheus flag?"
```

The most interesting here seems to be this error:

```
found multiple scrape configs with job name "PodMonitoring/keycloak/keycloak/http"
```

I assume that the logic of generating job names doesn't consider endpoint paths, and there must be no endpoints with the same port name. Is that correct?

As a workaround, I'll create one PodMonitoring resource per port.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.