actions / actions/actions-runner-controller
scheduledOverrides rely on only maxReplica value
Open
Nobody has claimed this yet.
bug
needs triage
- Dominant language
- Go
- Stars
- 6.5k
- Forks
- 1.5k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 27
Description
Checks
- I've already read https://github.com/actions/actions-runner-controller/blob/master/TROUBLESHOOTING.md and I'm sure my issue is not covered in the troubleshooting guide.
- I'm not using a custom entrypoint in my runner image
Controller Version
v0.27.0
Helm Chart Version
0.23.1
CertManager Version
1.5.3
Deployment Method
Helm
cert-manager installation
Cert manager was installed using the official helm chart from the jet stack repository.
Checks
- This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
- I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
- My actions-runner-controller version (v0.x.y) does support the feature
- I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue
- I've migrated to the workflow job webhook event (if you using webhook driven scaling)
Resource Definitions
$ kubectl get hra gha-staging-2c-8gb-autoscaler -n staging-2c-8gb -o yaml
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
annotations:
meta.helm.sh/release-name: staging-2c-8gb
meta.helm.sh/release-namespace: staging-2c-8gb
creationTimestamp: "2022-10-26T17:57:41Z"
generation: 448686
labels:
app.kubernetes.io/managed-by: Helm
name: gha-staging-2c-8gb-autoscaler
namespace: staging-2c-8gb
resourceVersion: "138167879"
uid: 446c274d-bebf-433b-a9e2-8da7defc3049
spec:
capacityReservations:
- effectiveTime: "2023-04-13T05:40:14Z"
expirationTime: "2023-04-13T17:40:14Z"
replicas: 1
- effectiveTime: "2023-04-13T05:40:18Z"
expirationTime: "2023-04-13T17:40:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:40:18Z"
expirationTime: "2023-04-13T17:40:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:40:18Z"
expirationTime: "2023-04-13T17:40:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:40:18Z"
expirationTime: "2023-04-13T17:40:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:40:18Z"
expirationTime: "2023-04-13T17:40:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:40:18Z"
expirationTime: "2023-04-13T17:40:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:40:18Z"
expirationTime: "2023-04-13T17:40:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:40:18Z"
expirationTime: "2023-04-13T17:40:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:40:18Z"
expirationTime: "2023-04-13T17:40:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:40:18Z"
expirationTime: "2023-04-13T17:40:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:40:18Z"
expirationTime: "2023-04-13T17:40:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:40:18Z"
expirationTime: "2023-04-13T17:40:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:40:18Z"
expirationTime: "2023-04-13T17:40:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:40:18Z"
expirationTime: "2023-04-13T17:40:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:40:18Z"
expirationTime: "2023-04-13T17:40:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:40:18Z"
expirationTime: "2023-04-13T17:40:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:40:18Z"
expirationTime: "2023-04-13T17:40:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:40:18Z"
expirationTime: "2023-04-13T17:40:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:40:18Z"
expirationTime: "2023-04-13T17:40:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:40:18Z"
expirationTime: "2023-04-13T17:40:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:40:18Z"
expirationTime: "2023-04-13T17:40:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:40:18Z"
expirationTime: "2023-04-13T17:40:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:40:18Z"
expirationTime: "2023-04-13T17:40:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:40:18Z"
expirationTime: "2023-04-13T17:40:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:40:18Z"
expirationTime: "2023-04-13T17:40:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:40:18Z"
expirationTime: "2023-04-13T17:40:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:40:18Z"
expirationTime: "2023-04-13T17:40:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:40:18Z"
expirationTime: "2023-04-13T17:40:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:40:18Z"
expirationTime: "2023-04-13T17:40:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:40:18Z"
expirationTime: "2023-04-13T17:40:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:40:18Z"
expirationTime: "2023-04-13T17:40:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:40:18Z"
expirationTime: "2023-04-13T17:40:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:40:18Z"
expirationTime: "2023-04-13T17:40:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:40:18Z"
expirationTime: "2023-04-13T17:40:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:40:18Z"
expirationTime: "2023-04-13T17:40:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:40:18Z"
expirationTime: "2023-04-13T17:40:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:40:18Z"
expirationTime: "2023-04-13T17:40:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:40:18Z"
expirationTime: "2023-04-13T17:40:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:40:18Z"
expirationTime: "2023-04-13T17:40:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:40:18Z"
expirationTime: "2023-04-13T17:40:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:40:18Z"
expirationTime: "2023-04-13T17:40:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:40:18Z"
expirationTime: "2023-04-13T17:40:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:40:18Z"
expirationTime: "2023-04-13T17:40:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:46:10Z"
expirationTime: "2023-04-13T17:46:10Z"
replicas: 1
- effectiveTime: "2023-04-13T05:46:17Z"
expirationTime: "2023-04-13T17:46:17Z"
replicas: 1
- effectiveTime: "2023-04-13T05:46:17Z"
expirationTime: "2023-04-13T17:46:17Z"
replicas: 1
- effectiveTime: "2023-04-13T05:46:44Z"
expirationTime: "2023-04-13T17:46:44Z"
replicas: 1
- effectiveTime: "2023-04-13T05:47:15Z"
expirationTime: "2023-04-13T17:47:15Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:14Z"
expirationTime: "2023-04-13T17:51:14Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
- effectiveTime: "2023-04-13T05:51:18Z"
expirationTime: "2023-04-13T17:51:18Z"
replicas: 1
maxReplicas: 50
minReplicas: 5
scaleDownDelaySecondsAfterScaleOut: 300
scaleTargetRef:
kind: RunnerDeployment
name: gha-staging-2c-8gb
scaleUpTriggers:
- duration: 12h
githubEvent:
workflowJob: {}
scheduledOverrides:
- endTime: "2023-02-27T00:00:00+01:00"
minReplicas: 1
recurrenceRule:
frequency: Weekly
startTime: "2023-02-25T00:00:00+01:00"
- endTime: "2023-02-28T18:00:00+01:00"
minReplicas: 10
recurrenceRule:
frequency: Daily
startTime: "2023-02-28T08:00:00+01:00"
status:
desiredReplicas: 50
lastSuccessfulScaleOutTime: "2023-04-13T05:45:35Z"
scheduledOverridesSummary: min=10 time=2023-04-13 07:00:00 +0000 UTC
To Reproduce
1. $ kubectl edit hra <hra> -n namespaces
2. Change maxReplica
Describe the bug
desiredReplicas is changing with the same value that we set in maxReplica. For instance, if I set maxReplica:50 then somehow desiredReplica updated itself with the same value as maxReplica without respecting scheduledOverrides rules.
$ kubectl get hra --all-namespaces
NAMESPACE NAME MIN MAX DESIRED SCHEDULE
staging-2c-8gb gha-staging-2c-8gb-autoscaler 5 50 50 min=10 time=2023-04-13 08:00:00 +0000 U
$ kubectl logs -f actions-runner-controller-c8456f7d7-jsz8x -n actions-runner-system | grep "staging-2c-8gb"
2023-04-13T06:25:16Z DEBUG horizontalrunnerautoscaler Checking scheduled override {"horizontalrunnerautoscaler": "staging-2c-8gb/gha-staging-2c-8gb-autoscaler", "now": "2023-04-13T06:25:16Z", "startTime": "2023-02-24 23:00:00 +0000 UTC", "endTime": "2023-02-26 23:00:00 +0000 UTC", "frequency": "Weekly", "untilTime": "0001-01-01 00:00:00 +0000 UTC"}
2023-04-13T06:25:16Z DEBUG horizontalrunnerautoscaler Found upcoming scheduled override {"horizontalrunnerautoscaler": "staging-2c-8gb/gha-staging-2c-8gb-autoscaler", "upcomingStartTime": "2023-04-14T23:00:00Z", "upcomingEndTime": "2023-04-16T23:00:00Z", "upcomingMinReplicas": 1}
2023-04-13T06:25:16Z DEBUG horizontalrunnerautoscaler Checking scheduled override {"horizontalrunnerautoscaler": "staging-2c-8gb/gha-staging-2c-8gb-autoscaler", "now": "2023-04-13T06:25:16Z", "startTime": "2023-03-12 08:00:00 +0000 UTC", "endTime": "2023-03-12 17:00:00 +0000 UTC", "frequency": "Daily", "untilTime": "0001-01-01 00:00:00 +0000 UTC"}
2023-04-13T06:25:16Z DEBUG horizontalrunnerautoscaler Found upcoming scheduled override {"horizontalrunnerautoscaler": "staging-2c-8gb/gha-staging-2c-8gb-autoscaler", "upcomingStartTime": "2023-04-13T08:00:00Z", "upcomingEndTime": "2023-04-13T17:00:00Z", "upcomingMinReplicas": 10}
2023-04-13T06:25:16Z DEBUG horizontalrunnerautoscaler Calculated desired replicas of 50 {"horizontalrunnerautoscaler": "staging-2c-8gb/gha-staging-2c-8gb-autoscaler", "suggested": 5, "reserved": 99, "min": 5, "max": 50}
Describe the expected behavior
scheduledOverrides should work properly.
Whole Controller Logs
<script src="https://gist.github.com/semihural-tomtom/e26cff1b7c73a5abbefb8e407b9e0ade.js"></script>
Whole Runner Pod Logs
The issue is related with schedule override
Additional Context
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the scheduledOverrides behavior from the supplied HorizontalRunnerAutoscaler YAML and trace the controller's scaling path. Confirm how maxReplica is used and determine the expected handling of the listed replica overrides. Done means scheduled overrides are evaluated correctly rather than relying only on maxReplica, with coverage for the reported resource shape.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, helm, kubernetes
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100