cockroachdb / cockroachdb/cockroach

roachprod: GC stale Prometheus targets

Open
#138,776 1 comment 0 reactions 0 assignees View on GitHub
A-testeng-infra A-testing C-enhancement T-testeng
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

When a `roachprod` cluster is deleted, the corresponding targets are deleted from `Prometheus` via the `prom-helper-service` API.

This has been known to be a flaky mechanism prone to deletion errors (e.g. #138772), which results in stale targets left in the Prometheus configuration and, with a combination of IP addresses recycling, some targets being scrapped multiple times, and sometimes scrapped insecurely (via plaintext HTTP) on HTTPS ports, which pollutes the cockroach.log file.

To avoid any future occurrence, a garbage collection mechanism should be considered to avoid drifts in the `Prometheus` configuration. This GC mechanism should be plugged in the `roachprod gc` command, which builds an exhaustive list of all living `roachprod` clusters.

Two possible logics are envisioned:
1. a quick and fast approach:
a. `roachprod` runs `cloud.ListCloud()` to build an exhaustive list of all `roachprod` clusters
b. `roachprod` calls a new `POST prom-helper-service/v1/instance-configs/clean-targets-outside-of-this-list` with the list of all living clusters as a body
3. an approach a bit more RESTfull:
a. `roachprod` runs `cloud.ListCloud()` to build an exhaustive list of all `roachprod` clusters
b. `roachprod` calls a new `GET prom-helper-service/v1/instance-configs/` endpoint, which returns all `Prometheus` targets
c. `roachprod` calls the `DELETE prom-helper/service/v1/instance-configs/${cluster}` for each stale configuration

Jira issue: CRDB-46344

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.