Workflow containers to run for runners in K8s mode take minutes to start up

We are running workflows on self-hosted runners in Kubernetes mode. The workflow is executed successfully, but it spends more than 3 minutes in the Initialise containers step. We have been able to detect the cause of the slowdown to be the action of copying files across from the runner to the pod defined in the PodTemplate. In fact the step took just 4-5 seconds when we removed the calls await copyExternalsToRoot() and await isAuthPermissionsOK() and replaced them with a bash copy command cp -R /home/runner/externals /home/runner/_work/externals before running /home/runner/run.sh in the Dockerfile image created based on the runner version ghcr.io/actions/actions-runner:2.317.0 where we only add a user with ID 1000 and make the necessary changes for the above copy command. Our cluster is in AWS and uses EFS as PersistentStorageType in ReadWriteMany access mode for the work volume shared by the Runner and the workflow pod.

We believe that the approach K8s mode without PV [DRAFT] #160 could solve this problem even better than our approach replacing the way to copy those files. I'm opening this ticket to keep track of the issue, and to signal that it's actually affecting us.

Start with packages/k8s/src/hooks/prepare-job.ts at copyExternalsToRoot() and packages/k8s/src/index.ts at isAuthPermissionsOK(), then compare the K8s mode without PV draft in PR #160. Reproduce initialization with an EFS-backed shared work volume; done means the Initialise containers step no longer takes several minutes while workflow execution remains successful.