SeleniumHQ / SeleniumHQ/docker-selenium
[🐛 Bug]: Nodes couldn't active when enabling autoscaling and deployed on EKS
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 8.7k
- Forks
- 2.5k
- Avg merge
- 10h 16m
- Merged PRs (30d)
- 20
Description
What happened?
When we enable autoscaling in helm chart, It doesn't work properly.
I'm using selenium grid helm chart on EKS. It works without autoscaling enabled But when I enable autoscaling, I couldn't see any active nodes in selenium.
Command used to start Selenium Grid with Docker (or Kubernetes)
value.yml for helm charts
hub:
serviceType: NodePort
autoscaling:
enabled: true
ingress:
enabled: true
nginx: !
annotations:
"kubernetes.io/ingress.class": "alb"
"alb.ingress.kubernetes.io/scheme": "internal"
"alb.ingress.kubernetes.io/group.name": "alb-name"
"alb.ingress.kubernetes.io/group.order": "300"
"alb.ingress.kubernetes.io/listen-ports": "[{\"HTTPS\":443}, {\"HTTP\":80}]"
"alb.ingress.kubernetes.io/ssl-redirect": "443"
"alb.ingress.kubernetes.io/healthcheck-port": "8080"
"alb.ingress.kubernetes.io/certificate-arn": "certificate-arn"
Relevant log output
kubectl logs keda-operator-bf9546dd-km68s
...
2024-04-28T18:48:30Z ERROR cert-rotation Webhook not found. Unable to update certificate. {"name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration", "error": "ValidatingWebhookConfiguration.admissionregistration.k8s.io \"keda-admission\" not found"}
github.com/open-policy-agent/cert-controller/pkg/rotator.(*ReconcileWH).ensureCerts
/workspace/vendor/github.com/open-policy-agent/cert-controller/pkg/rotator/rotator.go:816
github.com/open-policy-agent/cert-controller/pkg/rotator.(*ReconcileWH).Reconcile
/workspace/vendor/github.com/open-policy-agent/cert-controller/pkg/rotator/rotator.go:785
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227
2024-04-28T18:48:30Z INFO cert-rotation Ensuring CA cert {"name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService", "name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService"}
2024-04-28T18:48:30Z INFO cert-rotation no cert refresh needed
2024-04-28T18:48:30Z ERROR cert-rotation Webhook not found. Unable to update certificate. {"name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration", "error": "ValidatingWebhookConfiguration.admissionregistration.k8s.io \"keda-admission\" not found"}
github.com/open-policy-agent/cert-controller/pkg/rotator.(*ReconcileWH).ensureCerts
/workspace/vendor/github.com/open-policy-agent/cert-controller/pkg/rotator/rotator.go:816
github.com/open-policy-agent/cert-controller/pkg/rotator.(*ReconcileWH).Reconcile
/workspace/vendor/github.com/open-policy-agent/cert-controller/pkg/rotator/rotator.go:785
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227
2024-04-28T18:48:30Z INFO cert-rotation Ensuring CA cert {"name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService", "name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService"}
2024-04-28T18:48:32Z INFO cert-rotation CA certs are injected to webhooks
...
2024-04-28T18:48:42Z ERROR scaleexecutor failed to patch Objects {"scaledJob.Name": "selenium-chrome-node", "scaledJob.Namespace": "selenium", "error": "client rate limiter Wait returned an error: context canceled"}
github.com/kedacore/keda/v2/pkg/status.TransformObject
/workspace/pkg/status/status.go:195
github.com/kedacore/keda/v2/pkg/scaling/executor.(*scaleExecutor).setCondition
/workspace/pkg/scaling/executor/scale_executor.go:106
github.com/kedacore/keda/v2/pkg/scaling/executor.(*scaleExecutor).setActiveCondition
/workspace/pkg/scaling/executor/scale_executor.go:120
github.com/kedacore/keda/v2/pkg/scaling/executor.(*scaleExecutor).RequestJobScale
/workspace/pkg/scaling/executor/scale_jobs.go:76
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
/workspace/pkg/scaling/scale_handler.go:263
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
/workspace/pkg/scaling/scale_handler.go:182
2024-04-28T18:48:42Z ERROR scaleexecutor Error setting active condition when triggers are not active {"scaledJob.Name": "selenium-chrome-node", "scaledJob.Namespace": "selenium", "error": "client rate limiter Wait returned an error: context canceled"}
github.com/kedacore/keda/v2/pkg/scaling/executor.(*scaleExecutor).RequestJobScale
/workspace/pkg/scaling/executor/scale_jobs.go:77
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
/workspace/pkg/scaling/scale_handler.go:263
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
/workspace/pkg/scaling/scale_handler.go:182
...
2024-04-28T18:48:44Z ERROR scaleexecutor failed to patch Objects {"scaledJob.Name": "selenium-edge-node", "scaledJob.Namespace": "selenium", "error": "client rate limiter Wait returned an error: context canceled"}
github.com/kedacore/keda/v2/pkg/status.TransformObject
/workspace/pkg/status/status.go:195
github.com/kedacore/keda/v2/pkg/scaling/executor.(*scaleExecutor).setCondition
/workspace/pkg/scaling/executor/scale_executor.go:106
github.com/kedacore/keda/v2/pkg/scaling/executor.(*scaleExecutor).setActiveCondition
/workspace/pkg/scaling/executor/scale_executor.go:120
github.com/kedacore/keda/v2/pkg/scaling/executor.(*scaleExecutor).RequestJobScale
/workspace/pkg/scaling/executor/scale_jobs.go:76
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
/workspace/pkg/scaling/scale_handler.go:263
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
/workspace/pkg/scaling/scale_handler.go:182
2024-04-28T18:48:44Z ERROR scaleexecutor Error setting active condition when triggers are not active {"scaledJob.Name": "selenium-edge-node", "scaledJob.Namespace": "selenium", "error": "client rate limiter Wait returned an error: context canceled"}
github.com/kedacore/keda/v2/pkg/scaling/executor.(*scaleExecutor).RequestJobScale
/workspace/pkg/scaling/executor/scale_jobs.go:77
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
/workspace/pkg/scaling/scale_handler.go:263
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
/workspace/pkg/scaling/scale_handler.go:182
...
2024-04-28T18:48:45Z ERROR scaleexecutor failed to patch Objects {"scaledJob.Name": "selenium-firefox-node", "scaledJob.Namespace": "selenium", "error": "client rate limiter Wait returned an error: context canceled"}
github.com/kedacore/keda/v2/pkg/status.TransformObject
/workspace/pkg/status/status.go:195
github.com/kedacore/keda/v2/pkg/scaling/executor.(*scaleExecutor).setCondition
/workspace/pkg/scaling/executor/scale_executor.go:106
github.com/kedacore/keda/v2/pkg/scaling/executor.(*scaleExecutor).setActiveCondition
/workspace/pkg/scaling/executor/scale_executor.go:120
github.com/kedacore/keda/v2/pkg/scaling/executor.(*scaleExecutor).RequestJobScale
/workspace/pkg/scaling/executor/scale_jobs.go:76
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
/workspace/pkg/scaling/scale_handler.go:263
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
/workspace/pkg/scaling/scale_handler.go:182
2024-04-28T18:48:45Z ERROR scaleexecutor Error setting active condition when triggers are not active {"scaledJob.Name": "selenium-firefox-node", "scaledJob.Namespace": "selenium", "error": "client rate limiter Wait returned an error: context canceled"}
github.com/kedacore/keda/v2/pkg/scaling/executor.(*scaleExecutor).RequestJobScale
/workspace/pkg/scaling/executor/scale_jobs.go:77
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
/workspace/pkg/scaling/scale_handler.go:263
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
/workspace/pkg/scaling/scale_handler.go:182
Operating System
Kubernetes, EKS
Docker Selenium version (image tag)
4.20.0-20240425
Selenium Grid chart version (chart version)
0.30.0
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the issue with Selenium Grid chart 0.30.0, image 4.20.0-20240425, the supplied values, and EKS. Inspect the chart's autoscaling and KEDA resources, then compare their status with the KEDA logs and the reported absence of active nodes. Done means autoscaling creates active Selenium nodes without the reported errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, helm, kubernetes
- Domain
- cloud, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100