pingcap / pingcap/tidb-operator

Remove Cluster label in TiMonitor's prometheus scrape job.

Open
#4,219 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
1.3k
Forks
540
Avg merge
3d 2h
Merged PRs (30d)
18

Description

Backgroud

In my multi-kubernetes monitoring setup, we are using cluster label as external_labels to indicate an unique kubernetes cluster(I guess there are some more people doing the same🤔), when deploying timonitor and send its metrics to our central monitoring storage, I find some confusing results, cluster label are setted as tidb cluster.

Information I found

After digging into the impl, I find there are two kinds of cluster label providing almost the same information:

  1. cluster label:
    https://github.com/pingcap/tidb-operator/blob/d4c238c3d36377487f2225b627fc38a429e5c219/pkg/monitor/monitor/template.go#L375-L380

  2. tidb_cluster label
    https://github.com/pingcap/tidb-operator/blob/d4c238c3d36377487f2225b627fc38a429e5c219/pkg/monitor/monitor/template.go#L396-L402

Thus, I think it is okay to leave only one of them, namely tidb_cluster. This way, also avoiding the conflict I mentioned above.

Suggestions

I think we can first provide a config to disable this relabel setting for the compatiblity, and fianlly remove it after few versions later when we community reach concensus.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in pkg/monitor/monitor/template.go at the linked sections around lines 375-380 and 396-402, and compare how the two cluster labels are added to the Prometheus scrape job. Review the compatibility suggestion before deciding the configuration scope. Done means avoiding the conflicting cluster label while retaining tidb_cluster, with the transition behavior agreed by the community.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes, prometheus
Domain
observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.