occasionally `docker run` will hang for 30s-3min

Checks

• I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• I am using charts that are officially provided

Controller Version

0.23.5

Deployment Method

Helm

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions).
• I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

- Deploy a self hosted runner to kubernetes. I am using `dockerdWithinRunnerContainer: true` but I believe the same thing will happen without that


apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
spec:
  replicas: 1
  template:
    spec:
      dockerdWithinRunnerContainer: true
      repository: ...
      serviceAccountName: ...
      securityContext:
        # For Ubuntu 20.04 runner
        fsGroup: 1000
      resources:
        limits:
          memory: 20Gi
        requests:
          cpu: "6"
          memory: 20Gi

Describe the bug

Occasionally I see docker run commands take 2-3 minutes to start executing. This is true even if I pull the image first. For example, if a workflow runs:

docker pull hello-world
echo "--- DONE PULLING ---"
time docker run hello-world

Then occasionally, maybe 5% of the time, I see the pull happen, followed by a long delay, and then the run will finally execute. This can take 2-3 minutes to happen.

I can manually recreate this by exec-ing into the k8s pod using 2 shells and running this in each:

while true; do
    time docker run hello-world
done

Initially this will run quite fast, but at some point you'll see that the runtime will start to take upwards of 2-3 minutes occasionally. This seems to be some contention, but I cannot identify where the contention exists.

Describe the expected behavior

Docker run should happen quickly and consistently.

Additional Context

Nothing of note here

Controller Logs

Nothing of note here

Runner Pod Logs

Nothing of note here

Start with the RunnerDeployment manifest, especially dockerdWithinRunnerContainer, and reproduce the delay in the runner pod using two concurrent docker run hello-world loops. Check the controller logs and runner pod logs while comparing normal and delayed runs. Done means the intermittent 30-second-to-three-minute delay is explained and Docker runs start quickly and consistently.