actions / actions/actions-runner-controller
gha-runner-scale-set: the listenerMetrics example in values.yaml reads as a default but adds unbounded job_workflow_ref/name/target labels
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 6.5k
- Forks
- 1.5k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 27
Description
Checks
- I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
- I am using charts that are officially provided
Controller Version
gha-runner-scale-set 0.14.2 (latest release) and master at 54147cf; same block in gha-runner-scale-set-experimental.
Deployment Method
Helm
Checks
- This isn't a question or user support case (For Q&A and community support, go to Discussions).
- I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
To Reproduce
1. Enable listener metrics on the controller chart (`metrics.listenerAddr: ":8080"`, `metrics.listenerEndpoint: "/metrics"`).
2. In `charts/gha-runner-scale-set/values.yaml`, follow the instruction in the `listenerMetrics` comment ("When configuring metrics, please uncomment the listenerMetrics object below ... specify custom buckets for histogram"): uncomment the block as-is and change only the `buckets` list.
3. Run jobs from a handful of branches/PRs and scrape the listener.
4. Compare the label set of `gha_job_startup_duration_seconds_bucket` with a listener that has `listenerMetrics` unset.
Describe the bug
The comment above the example says the block is the defaults, shown for documentation:
## listenerMetrics are configurable metrics applied to the listener.
## In order to avoid helm merging these fields, we left the metrics commented out.
## When configuring metrics, please uncomment the listenerMetrics object below.
## You can modify the configuration to remove the label or specify custom buckets for histogram.
##
## If the buckets field is not specified, the default buckets will be applied. Default buckets are
## provided here for documentation purposes
But the label lists in the example are not the defaults. The example puts job_workflow_ref, job_workflow_name and job_workflow_target on both counters and both histograms (values.yaml L160, L170-L172, L194, L252-L254), while defaultMetrics in cmd/ghalistener/metrics/metrics.go#L162-L281 has none of them:
| metric | defaultMetrics labels |
extra labels in the values.yaml example |
|---|---|---|
gha_started_jobs_total |
enterprise, organization, repository, job_name, event_name | job_workflow_ref, job_workflow_name, job_workflow_target |
gha_completed_jobs_total |
+ job_result | job_workflow_ref, job_workflow_name, job_workflow_target |
gha_job_startup_duration_seconds |
enterprise, organization, repository, job_name, event_name | job_workflow_ref, job_workflow_name, job_workflow_target |
gha_job_execution_duration_seconds |
+ job_result | job_workflow_ref, job_workflow_name, job_workflow_target |
| all gauges | identical | none |
(Diffed by parsing the uncommented example and the defaultMetrics literal on master.)
job_workflow_ref was removed from the defaults in #3671 precisely because of cardinality (#3670, #3153) and re-introduced in #4054 only as an opt-in label plus this example; #4240 then added job_workflow_name/job_workflow_target to the example but again not to the defaults. So the example has drifted into a superset of the defaults while the comment still calls it the defaults.
The consequence for anyone who uncomments the block to tune buckets: job_workflow_ref carries the full ref (.../ci.yml@refs/pull/1234/merge, @refs/heads/<branch>) and job_workflow_target carries pull/1234 / the branch, so every PR or branch creates a new series set for each job_name, multiplied by 45 histogram buckets (+ _sum/_count). On a busy repository this is a few million series per histogram per week and histogram_quantile queries over the metric stop being usable. Nothing in the file warns that these two labels are per-ref.
Note: #4525 (open) edits the same comment block to explain where gauge vs job labels come from, but does not address the defaults/example mismatch or add a cardinality note; #4304 is about gauge label values and is separate.
Describe the expected behavior
- The example matches
defaultMetricsexactly, so uncommenting it to customise buckets does not change the label set; and/or the comment is corrected to say the example shows all available labels (a superset of the defaults) and points tocmd/ghalistener/metrics/metrics.gofor the actual defaults. - A short note next to the histograms:
job_workflow_refandjob_workflow_targetare per-ref/per-PR labels and each distinct value multiplies the series count by the number of buckets; preferjob_workflow_name, or keep them off the histograms.
Proposed fix (documentation only)
In charts/gha-runner-scale-set/values.yaml (L149-L256) and charts/gha-runner-scale-set-experimental/values.yaml (L315-L420):
- Replace "Default buckets are provided here for documentation purposes" with wording like: "The label lists below show every label the listener can emit. The built-in defaults (used when
listenerMetricsis unset) do NOT includejob_workflow_ref,job_workflow_nameorjob_workflow_target; seecmd/ghalistener/metrics/metrics.go(defaultMetrics)." - Either drop the three labels from the two histogram examples, or add above
histograms:a two-line warning that per-ref labels (job_workflow_ref,job_workflow_target) on histograms multiply cardinality by the bucket count and should only be enabled deliberately.
No code changes; ~10 comment lines per chart.
Additional Context
# gha-runner-scale-set values, exactly the shipped example uncommented, with only buckets changed
listenerMetrics:
histograms:
gha_job_startup_duration_seconds:
labels:
["repository", "organization", "enterprise", "job_name", "event_name", "job_workflow_ref", "job_workflow_name", "job_workflow_target"]
buckets: [1, 5, 10, 30, 60, 120, 300, 600]
gha_job_execution_duration_seconds:
labels:
["repository", "organization", "enterprise", "job_name", "event_name", "job_result", "job_workflow_ref", "job_workflow_name", "job_workflow_target"]
buckets: [10, 30, 60, 120, 300, 600, 1200, 1800, 3600]
Controller Logs
N/A - documentation/chart issue; controller behaviour is not involved. Listener scrape output shows the extra labels, e.g.:
gha_job_startup_duration_seconds_bucket{...,job_name="build",job_workflow_ref="org/repo/.github/workflows/ci.yml@refs/pull/1234/merge",job_workflow_target="pull/1234",le="5"} 1
Runner Pod Logs
N/A
Found while operating this in production; happy to send a PR if maintainers agree with the analysis.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with charts/gha-runner-scale-set/values.yaml and charts/gha-runner-scale-set-experimental/values.yaml, then compare their listenerMetrics examples with defaultMetrics in cmd/ghalistener/metrics/metrics.go. Update the chart comments and examples to accurately describe defaults and the cardinality implications of workflow labels. Done means both chart examples and their guidance no longer imply that opt-in labels are built-in defaults.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- helm
- Domain
- devops, documentation
- Issue type
- Documentation
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 88/100