Intermittently getting "Cannot connect to the Docker daemon at unix:///var/run/docker.sock"

Checks

• I've already read https://github.com/actions/actions-runner-controller/blob/master/TROUBLESHOOTING.md and I'm sure my issue is not covered in the troubleshooting guide.
• I'm not using a custom entrypoint in my runner image

Controller Version

0.9.3

Helm Chart Version

0.9.3

CertManager Version

1.16.1

Deployment Method

ArgoCD

cert-manager installation

cert-manager is working

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
• I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
• My actions-runner-controller version (v0.x.y) does support the feature
• I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue
• I've migrated to the workflow job webhook event (if you using webhook driven scaling)

Resource Definitions

values.yaml for our arc runner set helm installation:

githubConfigUrl: https://github.com/<org>
controllerServiceAccount:
  namespace: arc
  name: arc-gha-rs-controller
githubConfigSecret: arc-runner-set
maxRunners: 2
minRunners: 1
runnerGroup: "default"
runnerScaleSetName: "custom"
containerMode:
  type: dind
template:
  spec:
    hostNetwork: true
    containers:
    - name: runner
      image: some.azurecr.io/custom-actions-runner:latest
      command: ["/home/runner/run.sh"]
    imagePullSecrets:
    - name: acr-connectivity-pull
image:
  actionsRunnerImagePullSecrets:
  - name: acr-connectivity-pull

To Reproduce

Run any action which uses docker command, the error does not happen every time, I'd say it occurs 1/10 of the times, rerunning the job is usually successful.

Describe the bug

Running an action including docker command like:
docker build . --file Dockerfile --tag $env:FullImageName --secret id=npm_token,env=NPM_TOKEN --build-arg NODE_ENV=production
intermitently results in an error:

ERROR: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
NativeCommandExitException: /home/runner/_work/_temp/52c5c530-065c-45b1-b663-3abe54de30f1.ps1:5
Line |
5 | docker build . --file Dockerfile --tag $env:FullImageName --secret id …
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| Program "docker" ended with non-zero exit code: 1.

Describe the expected behavior

Being able to connect to unix:///var/run/docker.sock 100% of the runs.

Whole Controller Logs

https://gist.github.com/AurimasNav/398f849114ad71860eb0a0fcf465d691

Whole Runner Pod Logs

https://gist.github.com/AurimasNav/0660c09ba17d845591169ddf230dce48

Additional Context

In the dind container log I can see:

failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to register "bridge" driver: failed to create NAT chain DOCKER: iptables failed: iptables --wait -t nat -N DOCKER: iptables: Chain already exists.
(exit status 1)

Not sure why that happens or how it can be solved? Might this have something to do with my config in values.yaml

template:
  spec:
    hostNetwork: true

(if I don't specify this, my containers in actions have no internet access).

Start with TROUBLESHOOTING.md and the provided values.yaml, then review the linked controller and runner pod logs alongside the dind startup error. Reproduce a Docker action with the shown runner configuration and compare successful and failing runs. Done means the runner can connect to the Docker daemon reliably without the intermittent startup failure.