actions / actions/actions-runner-controller

HRA ignores webhook repository scale up events (but acks scaleDown)

Open
#999 6 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

documentation
Dominant language
Go
Stars
6.5k
Forks
1.5k
Avg merge
2d 2h
Merged PRs (30d)
27

Description

Describe the bug
It appears that under certain conditions, HRAs configured against a RunnerDeployment repository is not recognized on scaleUp events (but is recognized for scaleDown events)

Checks

  • My actions-runner-controller version (v0.x.y) does support the feature
  • I'm using an unreleased version of the controller I built from HEAD of the default branch

To Reproduce
Steps to reproduce the behavior:

  1. Create a repo-only PAT on an account that has member access to an organization repo where they are given the admin rights on the repository. (note: the user is not given admin access to the org)
  2. Configure webhook on the repo, setup actions controller, install PAT
  3. Create RunnerDeployment and HPA pointing at the repo
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: bar-runner
spec:
  template:
    spec:
      repository: foo/bar
      ## using eks / oidc
      # image: image:tag
      # serviceAccountName: svc-account-name
      # securityContext:
      #   fsGroup: 1000
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  name: bar-hra
spec:
  scaleTargetRef:
    name: bar-runner
  scaleUpTriggers:
  - githubEvent: {}
  minReplicas: 0
  maxReplicas: 10
  1. Trigger a workflow, webhook controller should identify the HRA (but fail to take action):
Found 1 HRAs by key     {"key": "foo/bar"}
Found 0 HRAs by key     {"key": "foo"}
no repository runner or organizational runner found     {"event": "workflow_job", "hookID": "332463031", "delivery": "37a4e5c0-591d-11ec-8c42-e1ea428feb2f", "workflowJob.status": "queued", "workflowJob.labels": ["self-hosted", "Linux", "x64"], "repository.name": "bar", "repository.owner.login": "foo", "repository.owner.type": "Organization", "enterprise.slug": "", "action": "queued", "repository": "foo/bar", "organization": "foo"}
Scale target not found. If this is unexpected, ensure that there is exactly one repository-wide or organizational runner deployment that matches this webhook event {"event": "workflow_job", "hookID": "332463031", "delivery": "37a4e5c0-591d-11ec-8c42-e1ea428feb2f", "workflowJob.status": "queued", "workflowJob.labels": ["self-hosted", "Linux", "x64"], "repository.name": "bar", "repository.owner.login": "foo", "repository.owner.type": "Organization", "enterprise.slug": "", "action": "queued"}
  1. Cancel the workflow, controller should both find the HRA and do something:
Found 1 HRAs by key     {"key": "foo/bar"}
job scale up target is repository-wide runners  {"event": "workflow_job", "hookID": "332463031", "delivery": "b68560f0-5921-11ec-8d7a-9f951ff061e6", "workflowJob.status": "completed", "workflowJob.labels": [], "repository.name": "bar", "repository.owner.login": "foo", "repository.owner.type": "Organization", "enterprise.slug": "", "action": "completed", "repository": "bar"}
Patching hra for capacityReservations update    {"before": null, "after": null}
scaled bar-hra by -1

Expected behavior
I expect the HRA to scale up the RD.

Screenshots
n/a

Environment (please complete the following information):

  • Helm Release - v0.15.1
  • eks - v1.21

Additional context
This is weird.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the webhook controller's HRA lookup and scale-up handling, comparing the repository key used for scaleUp with the working scaleDown path. Reproduce the issue with the provided RunnerDeployment and HorizontalRunnerAutoscaler manifests and workflow events; done means a repository-scoped HRA is recognized and its RunnerDeployment scales up on a queued job.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, go, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.