Runners keep throwing docker Daemon running

Checks

• I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• I am using charts that are officially provided

Controller Version

actions-runner-controller-0.22.0

Deployment Method

Helm

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions).
• I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

1) installed ARC with github token 
2) configured runnerdeployment  with replica -10
3) configured horizontal scaler - min 10 max 50
4) runners are in running state, but after some time new runners getting docker daemon running error and job are waiting in queue to pick the runner

happening for all the new runners then pod going to error state

Describe the bug

runners are in running state, but after some time new runners getting docker daemon running error and job are waiting in queue to pick the runner

happening for all the new runners then pod going to error state

Describe the expected behavior

based on the load horizontal scaler should scale the runners, but it is throwing docker daemon running ? error

Additional Context

❯ helm get values actions-runner-controller
USER-SUPPLIED VALUES:
authSecret:
  create: true
  github_token: "" supplied github token

Controller Logs

a2844b0833", "allowed": true}
2024-01-12T21:31:30Z	INFO	runner	Failed to create pod due to AlreadyExists error. Probably this pod has been already created in previous reconcilation but is still not in the informer cache. Will retry on pod created. If it doesn't repeat, there's no problem	{"runner": "actions-runner-systems/github-action-np-h6z4z-gf9pg"}
2024-01-12T21:31:31Z	DEBUG	runner	Runner appears to have been registered and running.	{"runner": "actions-runner-systems/github-action-np-h6z4z-gf9pg", "podCreationTimestamp": "2024-01-12 21:31:30 +0000 UTC"}
2024-01-12T21:31:36Z	INFO	runnerpod	Failed to delete pod within 1m0s. This is typically the case when a Kubernetes node became unreachable and the kube controller started evicting nodes. Forcefully deleting the pod to not get stuck.	{"runnerpod": "actions-runner-systems/github-action-np-h6z4z-qjc6z", "podDeletionTimestamp": "2024-01-12 21:30:25 +0000 UTC", "currentTime": "2024-01-12T21:31:36Z", "configuredDeletionTimeout": "1m0s"}
2024-01-12T21:31:36Z	INFO	runnerpod	Forcefully deleted runner pod	{"runnerpod": "actions-runner-systems/github-action-np-h6z4z-qjc6z", "repository": ""}
2024-01-12T21:31:36Z	DEBUG	events	Forcefully deleted pod 'github-action-np-h6z4z-qjc6z'	{"type": "Normal", "object": {"kind":"Pod","namespace":"actions-runner-systems","name":"github-action-np-h6z4z-qjc6z","uid":"3be68602-3db3-4803-a2b7-9ae0ec52df94","apiVersion":"v1","resourceVersion":"605720"}, "reason": "PodDeleted"}
2024-01-12T21:31:39Z	DEBUG	horizontalrunnerautoscaler	Suggested desired replicas of 10 by PercentageRunnersBusy	{"replicas_desired_before": 10, "replicas_desired": 10, "num_runners": 10, "num_runners_registered": 9, "num_runners_busy": 6, "num_terminating_busy": 0, "namespace": "actions-runner-systems", "kind": "runnerdeployment", "name": "github-action-np", "horizontal_runner_autoscaler": "example-runner-deployment-autoscaler", "enterprise": "", "organization": "prosperllc", "repository": ""}
2024-01-12T21:31:39Z	DEBUG	horizontalrunnerautoscaler	Calculated desired replicas of 10	{"horizontalrunnerautoscaler": "actions-runner-systems/example-runner-deployment-autoscaler", "suggested": 10, "reserved": 0, "min": 10, "max": 20}
2024-01-12T21:32:19Z	DEBUG	runner	Runner appears to have been registered and running.	{"runner": "actions-runner-systems/github-action-np-h6z4z-9f5cp", "podCreationTimestamp": "2024-01-12 21:25:58 +0000 UTC"}
2024-01-12T21:32:19Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/mutate-actions-summerwind-dev-v1alpha1-runner", "UID": "1e47c97c-345a-4b39-825f-67129cf2201d", "kind": "actions.summerwind.dev/v1alpha1, Kind=Runner", "resource": {"group":"actions.summerwind.dev","version":"v1alpha1","resource":"runners"}}
2024-01-12T21:32:19Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/mutate-actions-summerwind-dev-v1alpha1-runner", "code": 200, "reason": "", "UID": "1e47c97c-345a-4b39-825f-67129cf2201d", "allowed": true}
2024-01-12T21:32:19Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/mutate-actions-summerwind-dev-v1alpha1-runner", "UID": "89c6d115-f817-472c-b629-0489fd90e10e", "kind": "actions.summerwind.dev/v1alpha1, Kind=Runner", "resource": {"group":"actions.summerwind.dev","version":"v1alpha1","resource":"runners"}}
2024-01-12T21:32:19Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/mutate-actions-summerwind-dev-v1alpha1-runner", "code": 200, "reason": "", "UID": "89c6d115-f817-472c-b629-0489fd90e10e", "allowed": true}

Runner Pod Logs

"https://pipelinesghubeus21.actions.githubusercontent.com/tMTkzAKYleoidiHAI9FjPaHPkEkp2s7TIoUW3BW1740YmeFlFo/",
  "gitHubUrl": "https://github.com/prosperllc",
  "workFolder": "/runner/_work"
2024-01-12 21:34:47.510  DEBUG --- Docker enabled runner detected and Docker daemon wait is enabled
2024-01-12 21:34:47.512  DEBUG --- Waiting until Docker is available or the timeout of 120 seconds is reached
Failed to initialize: unable to resolve docker endpoint: open /certs/client/ca.pem: no such file or directory
Failed to initialize: unable to resolve docker endpoint: open /certs/client/ca.pem: no such file or directory
Failed to initialize: unable to resolve docker endpoint: open /certs/client/ca.pem: no such file or directory
Failed to initialize: unable to resolve docker endpoint: open /certs/client/ca.pem: no such file or directory
Cannot connect to the Docker daemon at tcp://localhost:2376. Is the docker daemon running?

Start with the Runner Pod Logs and the supplied Helm values, focusing on the missing /certs/client/ca.pem path and the Docker endpoint tcp://localhost:2376. Reproduce the autoscaled runner setup and trace the Docker daemon and certificate configuration; done means newly created runners start successfully and jobs leave the queue.