Disable high-cardinality tags in the metrics-server

What would you like added?

A way to stop the metrics-server from tagging metrics with tags of high cardinality.

Why is this needed?

The metrics-server currently tags metrics with high-cardinality tags like job_name, workflow_name, repository, head_branch (see https://github.com/actions/actions-runner-controller/blob/e0a7e142e0fcd446c58e7875d4d44a7eea6e72f2/pkg/actionsmetrics/metrics.go#L79). On top of that it publishes two distribution metrics which have about 45 buckets each (see https://github.com/actions/actions-runner-controller/blob/e0a7e142e0fcd446c58e7875d4d44a7eea6e72f2/pkg/actionsmetrics/metrics.go#L24). Since a separate value is publish for each combination of metric and tags, the number of published values grows very fast.

Example: If you have 10 different jobs names and 5 workflow names across 5 repositories with 5 branches each, you get 10*5*5*5=1250 different values. Since two metrics are distributions with 45 buckets, we get 2*45*1250=112500 values just from those two metrics.

Additional context

By default when you use Datadog to collect the metrics, it stops at the first 2000 values. The limit can be increased, but the data easily grows to several tens of MBs large per scrape.

If you let me know how you would like this configuration to work (e.g., a toggle per tag?) I'd be happy to open a PR for it.

Start with pkg/actionsmetrics/metrics.go, especially the tag definitions around the referenced line and the two distribution metrics with many buckets. Clarify whether configuration should disable individual tags or all high-cardinality tags; done means the metrics-server exposes that choice and no longer publishes disabled tag combinations.