cockroachdb / cockroachdb/cockroach
roachprod: GC stale Prometheus targets
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
When a `roachprod` cluster is deleted, the corresponding targets are deleted from `Prometheus` via the `prom-helper-service` API.
This has been known to be a flaky mechanism prone to deletion errors (e.g. #138772), which results in stale targets left in the Prometheus configuration and, with a combination of IP addresses recycling, some targets being scrapped multiple times, and sometimes scrapped insecurely (via plaintext HTTP) on HTTPS ports, which pollutes the cockroach.log file.
To avoid any future occurrence, a garbage collection mechanism should be considered to avoid drifts in the `Prometheus` configuration. This GC mechanism should be plugged in the `roachprod gc` command, which builds an exhaustive list of all living `roachprod` clusters.
Two possible logics are envisioned:
1. a quick and fast approach:
a. `roachprod` runs `cloud.ListCloud()` to build an exhaustive list of all `roachprod` clusters
b. `roachprod` calls a new `POST prom-helper-service/v1/instance-configs/clean-targets-outside-of-this-list` with the list of all living clusters as a body
3. an approach a bit more RESTfull:
a. `roachprod` runs `cloud.ListCloud()` to build an exhaustive list of all `roachprod` clusters
b. `roachprod` calls a new `GET prom-helper-service/v1/instance-configs/` endpoint, which returns all `Prometheus` targets
c. `roachprod` calls the `DELETE prom-helper/service/v1/instance-configs/${cluster}` for each stale configuration
Jira issue: CRDB-46344
Contributor guide
Assessment
This issue has not been assessed yet.