prometheus: reliable dynamic host discovery

While `roachprod` supported only single-tenant clusters, we could rely on `gce_sd_configs` to dynamically discover VMs and configure scrape targets on default ports, e.g., `26258` for CRDB metrics. However, as described in [1], the support for multi-tenant clusters added a new problem, namely the discovery of custom ports, assigned during the provisioning of (sql) tenants; i.e., `26258` remains as the `system` tenant, but other tenants are assigned random ports. (`roachprod.DiscoverService` maps tenants to their ports via DNS SRV records.)

The chosen solution (see the wiki linked in [1]) ended up using `file_sd_configs` backed by a simple REST API, implemented in [2]. In `roachprod.Start`, we invoke `UpdateTargets`, which instructs `prom-helper-service` to create `.yml` file on the prometheus host. E.g.,

```
head /opt/prom/prometheus/instance-configs/teamcity-18024845-1733294775-151-n6cpu4-geo.yml

- targets:
- 10.142.1.38:26258
labels:
cluster: teamcity-18024845-1733294775-151-n6cpu4-geo
host_ip: 10.142.1.38
instance: teamcity-18024845-1733294775-151-n6cpu4-geo-0003
job: cockroachdb
node: "3"
project: cockroach-ephemeral
region: us-east
```

In `roachprod.DestroyCluster`, we invoke `c, which instructs prom-helper-service` to remove the corresponding `.yml` file. This simple mechanism seems to work _assuming_ the invocations of `UpdateTargets` and `DeleteClusterConfig` succeed.

Since `roachprod.Start` can be invoked multiple times for a given cluster, e.g., starting a subset of the nodes at a time, `UpdateTargets` must be able to succeed each time; otherwise, it may fail to discover some of the tenants. Note, the current implementation doesn't even support tenants; it uses `system` instead.

Failing to execute `DeleteClusterConfig` results in a _stale_ scrape config. Because the labels are _static_, this can yield a rather undesirable side-effect, when the same private ip is being reused by an entirely different cluster. E.g., consider the following failure,

```
[w11] 2024/12/04 11:11:44 cluster_cloud.go:424: Failed to delete the cluster config with cluster as secure: DeleteClusterConfig: failed on url: https://grafana.testeng.crdb.io/promhelpers/v1/instance-configs/teamcity-18024999-1733294269-218-n6cpu4: Delete "https://grafana.testeng.crdb.io/promhelpers/v1/instance-configs/teamcity-18024999-1733294269-218-n6cpu4": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
```

At this point, the corresponding scrape config. will remain on the filesystem indefinitely. At a later time, a new cluster is going to reuse the same ip(s). Thus, the stale scrape config. is active again, except this time it's ingesting timeseries which are bogus, and duplicated.

While adding a GC service to `prom-helper-service` may seem like a solution for removing stale configs., it doesn't address the ip reuse. The labels should be _dynamically_ discovered from a VM, instead of statically assigned to an ip; it doesn't appear that `file_sd_configs` supports it.

[1] https://github.com/cockroachdb/cockroach/issues/117125
[2] https://github.com/cockroachlabs/prom-helper-service

Jira issue: CRDB-45248