SeleniumHQ / SeleniumHQ/docker-selenium

[🐛 Bug]: Nodes couldn't active when enabling autoscaling and deployed on EKS

Open
#2,232 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

I-autoscaling-k8s R-awaiting-answer
Dominant language
Go
Stars
8.7k
Forks
2.5k
Avg merge
10h 16m
Merged PRs (30d)
20

Description

What happened?

When we enable autoscaling in helm chart, It doesn't work properly.
I'm using selenium grid helm chart on EKS. It works without autoscaling enabled But when I enable autoscaling, I couldn't see any active nodes in selenium.
image

Command used to start Selenium Grid with Docker (or Kubernetes)
value.yml for helm charts
    hub:
      serviceType: NodePort

    autoscaling:
      enabled: true

    ingress:
      enabled: true
      nginx: !
      annotations:
        "kubernetes.io/ingress.class": "alb"
        "alb.ingress.kubernetes.io/scheme": "internal"
        "alb.ingress.kubernetes.io/group.name": "alb-name"
        "alb.ingress.kubernetes.io/group.order": "300"
        "alb.ingress.kubernetes.io/listen-ports": "[{\"HTTPS\":443}, {\"HTTP\":80}]"
        "alb.ingress.kubernetes.io/ssl-redirect": "443"
        "alb.ingress.kubernetes.io/healthcheck-port": "8080"
        "alb.ingress.kubernetes.io/certificate-arn": "certificate-arn"
Relevant log output
kubectl logs keda-operator-bf9546dd-km68s
...
2024-04-28T18:48:30Z    ERROR   cert-rotation   Webhook not found. Unable to update certificate.        {"name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration", "error": "ValidatingWebhookConfiguration.admissionregistration.k8s.io \"keda-admission\" not found"}
github.com/open-policy-agent/cert-controller/pkg/rotator.(*ReconcileWH).ensureCerts
        /workspace/vendor/github.com/open-policy-agent/cert-controller/pkg/rotator/rotator.go:816
github.com/open-policy-agent/cert-controller/pkg/rotator.(*ReconcileWH).Reconcile
        /workspace/vendor/github.com/open-policy-agent/cert-controller/pkg/rotator/rotator.go:785
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
        /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227
2024-04-28T18:48:30Z    INFO    cert-rotation   Ensuring CA cert        {"name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService", "name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService"}
2024-04-28T18:48:30Z    INFO    cert-rotation   no cert refresh needed
2024-04-28T18:48:30Z    ERROR   cert-rotation   Webhook not found. Unable to update certificate.        {"name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration", "error": "ValidatingWebhookConfiguration.admissionregistration.k8s.io \"keda-admission\" not found"}
github.com/open-policy-agent/cert-controller/pkg/rotator.(*ReconcileWH).ensureCerts
        /workspace/vendor/github.com/open-policy-agent/cert-controller/pkg/rotator/rotator.go:816
github.com/open-policy-agent/cert-controller/pkg/rotator.(*ReconcileWH).Reconcile
        /workspace/vendor/github.com/open-policy-agent/cert-controller/pkg/rotator/rotator.go:785
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
        /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227
2024-04-28T18:48:30Z    INFO    cert-rotation   Ensuring CA cert        {"name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService", "name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService"}
2024-04-28T18:48:32Z    INFO    cert-rotation   CA certs are injected to webhooks
...
2024-04-28T18:48:42Z    ERROR   scaleexecutor   failed to patch Objects {"scaledJob.Name": "selenium-chrome-node", "scaledJob.Namespace": "selenium", "error": "client rate limiter Wait returned an error: context canceled"}
github.com/kedacore/keda/v2/pkg/status.TransformObject
        /workspace/pkg/status/status.go:195
github.com/kedacore/keda/v2/pkg/scaling/executor.(*scaleExecutor).setCondition
        /workspace/pkg/scaling/executor/scale_executor.go:106
github.com/kedacore/keda/v2/pkg/scaling/executor.(*scaleExecutor).setActiveCondition
        /workspace/pkg/scaling/executor/scale_executor.go:120
github.com/kedacore/keda/v2/pkg/scaling/executor.(*scaleExecutor).RequestJobScale
        /workspace/pkg/scaling/executor/scale_jobs.go:76
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
        /workspace/pkg/scaling/scale_handler.go:263
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
        /workspace/pkg/scaling/scale_handler.go:182
2024-04-28T18:48:42Z    ERROR   scaleexecutor   Error setting active condition when triggers are not active     {"scaledJob.Name": "selenium-chrome-node", "scaledJob.Namespace": "selenium", "error": "client rate limiter Wait returned an error: context canceled"}
github.com/kedacore/keda/v2/pkg/scaling/executor.(*scaleExecutor).RequestJobScale
        /workspace/pkg/scaling/executor/scale_jobs.go:77
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
        /workspace/pkg/scaling/scale_handler.go:263
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
        /workspace/pkg/scaling/scale_handler.go:182
...
2024-04-28T18:48:44Z    ERROR   scaleexecutor   failed to patch Objects {"scaledJob.Name": "selenium-edge-node", "scaledJob.Namespace": "selenium", "error": "client rate limiter Wait returned an error: context canceled"}
github.com/kedacore/keda/v2/pkg/status.TransformObject
        /workspace/pkg/status/status.go:195
github.com/kedacore/keda/v2/pkg/scaling/executor.(*scaleExecutor).setCondition
        /workspace/pkg/scaling/executor/scale_executor.go:106
github.com/kedacore/keda/v2/pkg/scaling/executor.(*scaleExecutor).setActiveCondition
        /workspace/pkg/scaling/executor/scale_executor.go:120
github.com/kedacore/keda/v2/pkg/scaling/executor.(*scaleExecutor).RequestJobScale
        /workspace/pkg/scaling/executor/scale_jobs.go:76
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
        /workspace/pkg/scaling/scale_handler.go:263
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
        /workspace/pkg/scaling/scale_handler.go:182
2024-04-28T18:48:44Z    ERROR   scaleexecutor   Error setting active condition when triggers are not active     {"scaledJob.Name": "selenium-edge-node", "scaledJob.Namespace": "selenium", "error": "client rate limiter Wait returned an error: context canceled"}
github.com/kedacore/keda/v2/pkg/scaling/executor.(*scaleExecutor).RequestJobScale
        /workspace/pkg/scaling/executor/scale_jobs.go:77
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
        /workspace/pkg/scaling/scale_handler.go:263
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
        /workspace/pkg/scaling/scale_handler.go:182
...
2024-04-28T18:48:45Z    ERROR   scaleexecutor   failed to patch Objects {"scaledJob.Name": "selenium-firefox-node", "scaledJob.Namespace": "selenium", "error": "client rate limiter Wait returned an error: context canceled"}
github.com/kedacore/keda/v2/pkg/status.TransformObject
        /workspace/pkg/status/status.go:195
github.com/kedacore/keda/v2/pkg/scaling/executor.(*scaleExecutor).setCondition
        /workspace/pkg/scaling/executor/scale_executor.go:106
github.com/kedacore/keda/v2/pkg/scaling/executor.(*scaleExecutor).setActiveCondition
        /workspace/pkg/scaling/executor/scale_executor.go:120
github.com/kedacore/keda/v2/pkg/scaling/executor.(*scaleExecutor).RequestJobScale
        /workspace/pkg/scaling/executor/scale_jobs.go:76
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
        /workspace/pkg/scaling/scale_handler.go:263
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
        /workspace/pkg/scaling/scale_handler.go:182
2024-04-28T18:48:45Z    ERROR   scaleexecutor   Error setting active condition when triggers are not active     {"scaledJob.Name": "selenium-firefox-node", "scaledJob.Namespace": "selenium", "error": "client rate limiter Wait returned an error: context canceled"}
github.com/kedacore/keda/v2/pkg/scaling/executor.(*scaleExecutor).RequestJobScale
        /workspace/pkg/scaling/executor/scale_jobs.go:77
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
        /workspace/pkg/scaling/scale_handler.go:263
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
        /workspace/pkg/scaling/scale_handler.go:182
Operating System

Kubernetes, EKS

Docker Selenium version (image tag)

4.20.0-20240425

Selenium Grid chart version (chart version)

0.30.0

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the issue with Selenium Grid chart 0.30.0, image 4.20.0-20240425, the supplied values, and EKS. Inspect the chart's autoscaling and KEDA resources, then compare their status with the KEDA logs and the reported absence of active nodes. Done means autoscaling creates active Selenium nodes without the reported errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, helm, kubernetes
Domain
cloud, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.