GoogleCloudPlatform / GoogleCloudPlatform/prometheus-engine

rule-evaluator doesn't get updated alertmanager pod ipv4s

Open
#866 17 comments 2 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
232
Forks
109
Avg merge
1d 3h
Merged PRs (30d)
27

Description

hello - we're currently using Managed Prometheus and a self-hosted Alertmanager deployment. This has been functioning properly for over a year. We're currently on this version of `rule-evaluator`

```
gke.gcr.io/prometheus-engine/rule-evaluator:v0.8.1-gke.9
```

our `rule-evaluator` sends events to a self-managed Alertmanager statefulset, which lives in a separate namespace. We configure this via the `OperatorConfig` CRD:

```yaml
---
apiVersion: monitoring.googleapis.com/v1
kind: OperatorConfig
metadata:
namespace: gmp-public
name: config

# https://github.com/GoogleCloudPlatform/prometheus-engine/blob/main/doc/api.md#ruleevaluatorspec
rules:
alerting:
alertmanagers:
# configures where the rule-evaluator will send alerting events to
- name: alertmanager
namespace: monitoring
port: 9093

# https://github.com/GoogleCloudPlatform/prometheus-engine/blob/main/doc/api.md#managedalertmanagerspec
# NOTE: this section is unused, as it points to an empty default Alertmanager configuration file
# since we are using a self-deployed Alertmanager instead of the one provided by GMP
managedAlertmanager:
configSecret:
name: alertmanager
key: alertmanager.yaml

```

in the last month or so, we've noticed that the `rule-evaluator` will be unable to resolve the downstream Alertmanager address after the Alertmanager pod is rescheduled.

From there, we'll see the `rule-evaluator` log this out:

```
{
alertmanager: "http://10.34.25.18:9093/api/v2/alerts"
caller: "notifier.go:532"
component: "notifier"
count: 1
err: "Post "http://10.34.25.18:9093/api/v2/alerts": context deadline exceeded"
level: "error"
msg: "Error sending alert"
ts: "2024-02-12T18:17:33.124842335Z"
}
```

this can go on for an hour - we have pages set up to notify us when the `rule-evaluator` stops pinging Alertmanager through a custom heartbeat rule. The only way to resolve this is by restarting the `rule-evaluator` deployment

this suggests that the `rule-evaluator` is not reconciling downstream ip addresses after startup, since we provide the k8s DNS components in the `OperatorConfig` for the Alertmanager receiver

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.