actions / actions/actions-runner-controller

Webhook Based horizontalrunnerautoscaler can't use Additional Autoscaling Prometheus Metrics

Open
#2,799 4 comments 18 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug community needs triage
Dominant language
Go
Stars
6.5k
Forks
1.5k
Avg merge
2d 2h
Merged PRs (30d)
27

Description

Checks
Controller Version

0.27.4

Helm Chart Version

0.23.3

CertManager Version

1.7.1

Deployment Method

Helm

cert-manager installation

I installed cert-manager correctly by helm (v1.7.1)

Checks
  • This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
  • I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
  • My actions-runner-controller version (v0.x.y) does support the feature
  • I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue
  • I've migrated to the workflow job webhook event (if you using webhook driven scaling)
Resource Definitions
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  namespace: github-actions
  name: github-actions-hra
spec:
  minReplicas: 1
  maxReplicas: 20
  scaleTargetRef:
    kind: RunnerDeployment
    name: github-actions-runner
  scheduledOverrides:
  - startTime: "2023-07-26T00:00:00Z"
    endTime: "2023-07-26T12:00:00Z"
    recurrenceRule:
      frequency: Daily
    minReplicas: 2
  scaleUpTriggers:
    - githubEvent:
        workflowJob: {}
      duration: "60m"
To Reproduce
1. set `metrics.serviceMonitor.enabled` to `true`, `actionsMetricsServer.enabled` to `true`
2. install helm chart
3. check prometheus dashboard
4. can see only 3 HorizontalRunnerAutoscaler metrics.
Describe the bug

at first, I see this discussions.

I expected various promethus metrics for hra (because this PR is merged) , but I can see only 3 metrics. This means that the newly added 10 metric was not visible.

Looking at the discussion history, it seems that there are many users who experience the same issues as me.

https://github.com/actions/actions-runner-controller/blob/dcb64f0b9e889f4751bf257f2a37632ce10a3e10/controllers/actions.summerwind.net/autoscaling.go#L51-L58

After reviewing the internal code, it is determined that additional metrics are not available for webhook based hra.

Describe the expected behavior

Support Webhook Based horizontalrunnerautoscaler to use Additional Autoscaling Prometheus Metrics

  • horizontalrunnerautoscaler_replicas_desired
  • horizontalrunnerautoscaler_runners
  • horizontalrunnerautoscaler_runners_registered
  • horizontalrunnerautoscaler_runners_busy
  • horizontalrunnerautoscaler_terminating_busy
  • horizontalrunnerautoscaler_necessary_replicas
  • horizontalrunnerautoscaler_workflow_runs_completed
  • horizontalrunnerautoscaler_workflow_runs_in_progress
  • horizontalrunnerautoscaler_workflow_runs_queued
  • horizontalrunnerautoscaler_workflow_runs_unknown
Whole Controller Logs
.
Whole Runner Pod Logs
.
Additional Context

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading controllers/actions.summerwind.net/autoscaling.go around lines 51-58 and trace how metrics differ for webhook-based and other HorizontalRunnerAutoscaler paths. Reproduce with metrics.serviceMonitor.enabled and actionsMetricsServer.enabled in the Helm chart, then verify that all ten additional HRA metrics are exposed.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, helm, kubernetes, prometheus
Domain
backend, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.