actions / actions/actions-runner-controller

RunnerDeployment creates an uncontrolled number of Runners

Open
#875 12 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug help wanted
Dominant language
Go
Stars
6.5k
Forks
1.5k
Avg merge
2d 2h
Merged PRs (30d)
27

Description

Describe the bug
RunnerDeployment creates an uncontrolled number of Runners. Starting from helm chart version 0.13.0

Checks

  • [*] My actions-runner-controller version (v0.x.y) does support the feature

To Reproduce
Steps to reproduce the behavior:

  1. Install actions-runner-controller via helm chart (0.13.0, 0.13.1, 0.13.2 versions have this issue, 0.12.8 and below - don't)
    helm upgrade --install -n gh-actions actions-runner-controller actions-runner-controller/actions-runner-controller --set authSecret.create=true --set scope.watchNamespace=gh-actions --set scope.singleNamespace=true --set securityContext.runAsUser=999 --set authSecret.github_token=*** --version "0.13.1"
  2. Install RunnerDeployment kubectl -n gh-actions apply -f org-runner-deployment.yaml
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: runner-deployment-1
spec:
  replicas: 1
  template:
    spec:
      repository: existing-repo-name/existing-project-name
  1. Observe RunnerReplicaSet, Runner and Pod resources => Their amount grows without a stop. In my case I got about 25000 runners before I realized what's happening. Pods eventually cannot start by hitting request limits. There are so many github API requests happens that quota is reached and it becomes very difficult to cleanup all the created resources (because of the Runner finalizers)

Expected behavior
1 RunnerReplicaSet, 1 Runner and one Pod should be created

Environment (please complete the following information):

  • Controller Version 0.20.1
  • Deployment Method: Helm
  • Helm Chart Version: 0.13.1 (or 0.13.0, or 0.13.2)

Additional context
Controller logs:
gh-actions-actions-runner-controller-86c4cdfccc-tzxb5-1633593533072392200.log
List of runners after a couple of seconds (exported from k9s):
runners-gh-actions-1633593544480491500.csv

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the issue with Helm chart 0.13.1 and the supplied RunnerDeployment, then inspect the RunnerDeployment reconciliation path and compare it with chart 0.12.8. Done means the deployment maintains exactly one RunnerReplicaSet, Runner, and Pod for replicas: 1 without uncontrolled resource growth or excessive GitHub API requests.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, helm, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.