actions / actions/actions-runner-controller

gha-runner-scale-set: the listenerMetrics example in values.yaml reads as a default but adds unbounded job_workflow_ref/name/target labels

Open Beginner friendly
#4,625 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
6.5k
Forks
1.5k
Avg merge
2d 2h
Merged PRs (30d)
27

Description

Checks
Controller Version

gha-runner-scale-set 0.14.2 (latest release) and master at 54147cf; same block in gha-runner-scale-set-experimental.

Deployment Method

Helm

Checks
  • This isn't a question or user support case (For Q&A and community support, go to Discussions).
  • I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
To Reproduce
1. Enable listener metrics on the controller chart (`metrics.listenerAddr: ":8080"`, `metrics.listenerEndpoint: "/metrics"`).
2. In `charts/gha-runner-scale-set/values.yaml`, follow the instruction in the `listenerMetrics` comment ("When configuring metrics, please uncomment the listenerMetrics object below ... specify custom buckets for histogram"): uncomment the block as-is and change only the `buckets` list.
3. Run jobs from a handful of branches/PRs and scrape the listener.
4. Compare the label set of `gha_job_startup_duration_seconds_bucket` with a listener that has `listenerMetrics` unset.
Describe the bug

The comment above the example says the block is the defaults, shown for documentation:

https://github.com/actions/actions-runner-controller/blob/54147cfa5eda5d9bbd798b99d40bf3f4b4402e82/charts/gha-runner-scale-set/values.yaml#L149-L156

## listenerMetrics are configurable metrics applied to the listener.
## In order to avoid helm merging these fields, we left the metrics commented out.
## When configuring metrics, please uncomment the listenerMetrics object below.
## You can modify the configuration to remove the label or specify custom buckets for histogram.
##
## If the buckets field is not specified, the default buckets will be applied. Default buckets are
## provided here for documentation purposes

But the label lists in the example are not the defaults. The example puts job_workflow_ref, job_workflow_name and job_workflow_target on both counters and both histograms (values.yaml L160, L170-L172, L194, L252-L254), while defaultMetrics in cmd/ghalistener/metrics/metrics.go#L162-L281 has none of them:

metric defaultMetrics labels extra labels in the values.yaml example
gha_started_jobs_total enterprise, organization, repository, job_name, event_name job_workflow_ref, job_workflow_name, job_workflow_target
gha_completed_jobs_total + job_result job_workflow_ref, job_workflow_name, job_workflow_target
gha_job_startup_duration_seconds enterprise, organization, repository, job_name, event_name job_workflow_ref, job_workflow_name, job_workflow_target
gha_job_execution_duration_seconds + job_result job_workflow_ref, job_workflow_name, job_workflow_target
all gauges identical none

(Diffed by parsing the uncommented example and the defaultMetrics literal on master.)

job_workflow_ref was removed from the defaults in #3671 precisely because of cardinality (#3670, #3153) and re-introduced in #4054 only as an opt-in label plus this example; #4240 then added job_workflow_name/job_workflow_target to the example but again not to the defaults. So the example has drifted into a superset of the defaults while the comment still calls it the defaults.

The consequence for anyone who uncomments the block to tune buckets: job_workflow_ref carries the full ref (.../ci.yml@refs/pull/1234/merge, @refs/heads/<branch>) and job_workflow_target carries pull/1234 / the branch, so every PR or branch creates a new series set for each job_name, multiplied by 45 histogram buckets (+ _sum/_count). On a busy repository this is a few million series per histogram per week and histogram_quantile queries over the metric stop being usable. Nothing in the file warns that these two labels are per-ref.

Note: #4525 (open) edits the same comment block to explain where gauge vs job labels come from, but does not address the defaults/example mismatch or add a cardinality note; #4304 is about gauge label values and is separate.

Describe the expected behavior
  • The example matches defaultMetrics exactly, so uncommenting it to customise buckets does not change the label set; and/or the comment is corrected to say the example shows all available labels (a superset of the defaults) and points to cmd/ghalistener/metrics/metrics.go for the actual defaults.
  • A short note next to the histograms: job_workflow_ref and job_workflow_target are per-ref/per-PR labels and each distinct value multiplies the series count by the number of buckets; prefer job_workflow_name, or keep them off the histograms.
Proposed fix (documentation only)

In charts/gha-runner-scale-set/values.yaml (L149-L256) and charts/gha-runner-scale-set-experimental/values.yaml (L315-L420):

  1. Replace "Default buckets are provided here for documentation purposes" with wording like: "The label lists below show every label the listener can emit. The built-in defaults (used when listenerMetrics is unset) do NOT include job_workflow_ref, job_workflow_name or job_workflow_target; see cmd/ghalistener/metrics/metrics.go (defaultMetrics)."
  2. Either drop the three labels from the two histogram examples, or add above histograms: a two-line warning that per-ref labels (job_workflow_ref, job_workflow_target) on histograms multiply cardinality by the bucket count and should only be enabled deliberately.

No code changes; ~10 comment lines per chart.

Additional Context
# gha-runner-scale-set values, exactly the shipped example uncommented, with only buckets changed
listenerMetrics:
  histograms:
    gha_job_startup_duration_seconds:
      labels:
        ["repository", "organization", "enterprise", "job_name", "event_name", "job_workflow_ref", "job_workflow_name", "job_workflow_target"]
      buckets: [1, 5, 10, 30, 60, 120, 300, 600]
    gha_job_execution_duration_seconds:
      labels:
        ["repository", "organization", "enterprise", "job_name", "event_name", "job_result", "job_workflow_ref", "job_workflow_name", "job_workflow_target"]
      buckets: [10, 30, 60, 120, 300, 600, 1200, 1800, 3600]
Controller Logs
N/A - documentation/chart issue; controller behaviour is not involved. Listener scrape output shows the extra labels, e.g.:
gha_job_startup_duration_seconds_bucket{...,job_name="build",job_workflow_ref="org/repo/.github/workflows/ci.yml@refs/pull/1234/merge",job_workflow_target="pull/1234",le="5"} 1
Runner Pod Logs
N/A

Found while operating this in production; happy to send a PR if maintainers agree with the analysis.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with charts/gha-runner-scale-set/values.yaml and charts/gha-runner-scale-set-experimental/values.yaml, then compare their listenerMetrics examples with defaultMetrics in cmd/ghalistener/metrics/metrics.go. Update the chart comments and examples to accurately describe defaults and the cardinality implications of workflow labels. Done means both chart examples and their guidance no longer imply that opt-in labels are built-in defaults.

Written by the indexing model from the issue text.

Assessment

Tech stack
helm
Domain
devops, documentation
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
88/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.