actions / actions/actions-runner-controller
RunnerDeployment creates an uncontrolled number of Runners
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 6.5k
- Forks
- 1.5k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 27
Description
Describe the bug
RunnerDeployment creates an uncontrolled number of Runners. Starting from helm chart version 0.13.0
Checks
- [*] My actions-runner-controller version (v0.x.y) does support the feature
To Reproduce
Steps to reproduce the behavior:
- Install actions-runner-controller via helm chart (0.13.0, 0.13.1, 0.13.2 versions have this issue, 0.12.8 and below - don't)
helm upgrade --install -n gh-actions actions-runner-controller actions-runner-controller/actions-runner-controller --set authSecret.create=true --set scope.watchNamespace=gh-actions --set scope.singleNamespace=true --set securityContext.runAsUser=999 --set authSecret.github_token=*** --version "0.13.1" - Install RunnerDeployment
kubectl -n gh-actions apply -f org-runner-deployment.yaml
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
name: runner-deployment-1
spec:
replicas: 1
template:
spec:
repository: existing-repo-name/existing-project-name
- Observe
RunnerReplicaSet,RunnerandPodresources => Their amount grows without a stop. In my case I got about 25000 runners before I realized what's happening. Pods eventually cannot start by hitting request limits. There are so many github API requests happens that quota is reached and it becomes very difficult to cleanup all the created resources (because of the Runner finalizers)
Expected behavior
1 RunnerReplicaSet, 1 Runner and one Pod should be created
Environment (please complete the following information):
- Controller Version 0.20.1
- Deployment Method: Helm
- Helm Chart Version: 0.13.1 (or 0.13.0, or 0.13.2)
Additional context
Controller logs:
gh-actions-actions-runner-controller-86c4cdfccc-tzxb5-1633593533072392200.log
List of runners after a couple of seconds (exported from k9s):
runners-gh-actions-1633593544480491500.csv
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the issue with Helm chart 0.13.1 and the supplied RunnerDeployment, then inspect the RunnerDeployment reconciliation path and compare it with chart 0.12.8. Done means the deployment maintains exactly one RunnerReplicaSet, Runner, and Pod for replicas: 1 without uncontrolled resource growth or excessive GitHub API requests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, helm, kubernetes
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100