Docker daemon is not responding

Checks

• I've already read https://github.com/actions/actions-runner-controller/blob/master/TROUBLESHOOTING.md and I'm sure my issue is not covered in the troubleshooting guide.
• I'm not using a custom entrypoint in my runner image

Controller Version

0.27.4

Helm Chart Version

0.23.3

CertManager Version

1.10.0

Deployment Method

ArgoCD

cert-manager installation

Installed via Helm

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
• I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
• My actions-runner-controller version (v0.x.y) does support the feature
• I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue
• I've migrated to the workflow job webhook event (if you using webhook driven scaling)

Resource Definitions

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: runners
  namespace: actions-runner-system
spec:
  template:
    metadata:
      annotations:
        cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
    spec:
      dockerMTU: 1400
      image: ghcr.io/actions-runner-controller/actions-runner-controller/actions-runner:v2.304.0-ubuntu-20.04
      imagePullPolicy: "Always"
      ephemeral: true
      organization: "<hidden>"
      labels: [<hidden>]

To Reproduce

1. Run a job with Docker interaction
2. Randomly get

Error response from daemon: error creating temporary lease: connection error: desc = "transport: Error while dialing dial unix:///var/run/docker/containerd/containerd.sock: timeout": unavailable

or

Cannot connect to the Docker daemon at unix:///run/docker.sock. Is the docker daemon running?

or other error where Docker daemon is not responding

Describe the bug

Docker daemon is not responding

Describe the expected behavior

Docker daemon is responding

Whole Controller Logs

https://gist.github.com/Tarasovych/a888e8c7e1edebc26ca1c547a778a860

Whole Runner Pod Logs

https://gist.github.com/Tarasovych/759f79483e9295ff0ad3fd020bbe8459

Additional Context

Similar issue: https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2408#note_383326688
Disk space is enough for the runner, with 3000 IOPS and 200 throughput.
/run directory has enough disk space (~ 100 KB free space at the peak load).

Start with the Whole Controller Logs and Whole Runner Pod Logs gists, then compare them with the RunnerDeployment definition and the Docker daemon errors in the reproduction steps. Reproduce a Docker-based job under the listed runner configuration and trace the intermittent failure; done means the cause is identified and the runner's Docker daemon responds reliably.