Permission denied to cache persistent volume with runner-scale-set

Checks

• I've already read https://github.com/actions/actions-runner-controller/blob/master/TROUBLESHOOTING.md and I'm sure my issue is not covered in the troubleshooting guide.
• I'm not using a custom entrypoint in my runner image

Controller Version

0.5.0

Helm Chart Version

0.5.0

CertManager Version

No response

Deployment Method

Helm

cert-manager installation

N/A

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
• I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
• My actions-runner-controller version (v0.x.y) does support the feature
• I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue
• I've migrated to the workflow job webhook event (if you using webhook driven scaling)

Resource Definitions

apiVersion: actions.github.com/v1alpha1
kind: EphemeralRunnerSet
metadata:
  annotations:
    actions.github.com/runner-group-name: Default
  creationTimestamp: '2023-09-13T14:11:25Z'
  finalizers:
    - ephemeralrunner.actions.github.com/finalizer
  generateName: REDACTED-github-runner-
  generation: 3
  labels:
    actions.github.com/organization: REDACTED
    actions.github.com/scale-set-name: REDACTED-github-runner
    actions.github.com/scale-set-namespace: arc-runner
    app.kubernetes.io/component: runner-set
    app.kubernetes.io/part-of: gha-runner-scale-set
    app.kubernetes.io/version: 0.5.0
    runner-spec-hash: 67bbc678dc
  name: REDACTED-github-runner-gsm95
  namespace: arc-runner
  resourceVersion: '516167472'
  uid: bd1675ed-0942-4d67-85a1-2a242f959952
  selfLink: >-
    /apis/actions.github.com/v1alpha1/namespaces/arc-runner/ephemeralrunnersets/REDACTED-github-runner-gsm95
status:
  currentReplicas: 0
  pendingEphemeralRunners: 0
  runningEphemeralRunners: 0
spec:
  ephemeralRunnerSpec:
    githubConfigSecret: github-runner
    githubConfigUrl: https://github.com/REDACTED
    metadata: {}
    runnerScaleSetId: 1
    spec:
      containers:
        - command:
            - /home/runner/run.sh
          env:
            - name: DOCKER_HOST
              value: tcp://localhost:2376
            - name: DOCKER_TLS_VERIFY
              value: '1'
            - name: DOCKER_CERT_PATH
              value: /certs/client
            - name: GOMODCACHE
              value: /home/runner/.cache/go-mod
          image: >-
            CUSTOM-REGISTRY/github-actions-runner:latest
          name: runner
          resources:
            limits:
              cpu: '2'
              memory: 1Gi
            requests:
              cpu: '1'
              memory: 512Mi
          volumeMounts:
            - mountPath: /home/runner/_work
              name: work
            - mountPath: /certs/client
              name: dind-cert
              readOnly: true
            - mountPath: /home/runner/.cache
              name: cache
        - image: docker:dind
          name: dind
          resources: {}
          securityContext:
            privileged: true
          volumeMounts:
            - mountPath: /home/runner/_work
              name: work
            - mountPath: /certs/client
              name: dind-cert
            - mountPath: /home/runner/externals
              name: dind-externals
            - mountPath: /var/lib/docker
              name: var-lib-docker
      initContainers:
        - command:
            - cp
            - '-r'
            - '-v'
            - /home/runner/externals/.
            - /home/runner/tmpDir/
          image: ghcr.io/actions/actions-runner:latest
          name: init-dind-externals
          resources: {}
          volumeMounts:
            - mountPath: /home/runner/tmpDir
              name: dind-externals
      serviceAccountName: REDACTED-github-runner-gha-rs-no-permission
      volumes:
        - emptyDir: {}
          name: work
        - emptyDir: {}
          name: dind-cert
        - emptyDir: {}
          name: dind-externals
        - ephemeral:
            volumeClaimTemplate:
              metadata: {}
              spec:
                accessModes:
                  - ReadWriteOnce
                resources:
                  requests:
                    storage: 5Gi
                storageClassName: standard
          name: cache
        - ephemeral:
            volumeClaimTemplate:
              metadata: {}
              spec:
                accessModes:
                  - ReadWriteOnce
                resources:
                  requests:
                    storage: 5Gi
                storageClassName: standard
          name: var-lib-docker

To Reproduce

1. Deploy the gha-runner-scale-set-controller chart
2. Deploy the gha-runner-scale-set chart with the following values


template:
  spec:
    initContainers:
    - name: init-dind-externals
      image: ghcr.io/actions/actions-runner:latest
      command: ["cp", "-r", "-v", "/home/runner/externals/.", "/home/runner/tmpDir/"]
      volumeMounts:
        - name: dind-externals
          mountPath: /home/runner/tmpDir
    containers:
    - name: runner
      image: CUSTOM-REGISTRY/github-actions-runner:latest
      command: ["/home/runner/run.sh"]
      env:
        - name: DOCKER_HOST
          value: tcp://localhost:2376
        - name: DOCKER_TLS_VERIFY
          value: "1"
        - name: DOCKER_CERT_PATH
          value: /certs/client
        - name: GOMODCACHE
          value: "/home/runner/.cache/go-mod"
      volumeMounts:
        - name: work
          mountPath: /home/runner/_work
        - name: dind-cert
          mountPath: /certs/client
          readOnly: true
        - name: cache
          mountPath: /home/runner/.cache
    - name: dind
      image: docker:dind
      securityContext:
        privileged: true
      volumeMounts:
        - name: work
          mountPath: /home/runner/_work
        - name: dind-cert
          mountPath: /certs/client
        - name: dind-externals
          mountPath: /home/runner/externals
        - name: var-lib-docker
          mountPath: /var/lib/docker
    volumes:
    - name: work
      emptyDir: {}
    - name: dind-cert
      emptyDir: {}
    - name: dind-externals
      emptyDir: {}
    - name: cache
      ephemeral:
        volumeClaimTemplate:
          spec:
            accessModes: [ "ReadWriteOnce" ]
            storageClassName: "standard"
            resources:
              requests:
                storage: 5Gi
    - name: var-lib-docker
      ephemeral:
        volumeClaimTemplate:
          spec:
            accessModes: [ "ReadWriteOnce" ]
            storageClassName: "standard"
            resources:
              requests:
                storage: 5Gi

3. Run a job that fetches go dependencies. For example

- name: Set up Go build environment
  id: go
  uses: actions/setup-go@v4
  with:
    go-version: 1.19

- name: Get Go Dependencies
  shell: bash
  run: |
    go get -v -t -d ./...

### Describe the bug

When the job runs, the setup-go step successfully installs go and prints the correct path for `GOMODCACHE` and `GOCACHE`. When it runs `go get...` it returns the error

`go: github.com/REDACTED/REDACTED/v2@v2.4.26: mkdir /home/runner/.cache/go-mod: permission denied`

This was working until adding the `volumeClaimTemplates` for the cache.

We see similar errors with the `setup-helmfile` action:

  - name: Install helmfile
    uses: mamezou-tech/setup-helmfile@v1.2.0
    with:
      helmfile-version: "v0.151.0"
      kubectl-version: 1.25.12
      kubectl-release-date: "2023-08-16"
      helm-version: "v3.12.3"

Error: Unable to get repository: fatal: could not create leading directories of '/home/runner/.cache/helm/plugins/https-github.com-databus23-helm-diff': Permission denied
: exit status 128

### Describe the expected behavior

These actions should be able to write to the cache directory/PersistentVolume without errors.

There is very little documentation for the `gha-runner-scale-set` deployment model compared to the legacy ARC deployment, so I did my best to piece together the correct configuration from the legacy docs.

### Whole Controller Logs

```shell
https://gist.github.com/bkonicek-calm/6a83445110b208938d56858aa90eed0b

Whole Runner Pod Logs

Docker logs: https://gist.github.com/bkonicek-calm/f849356778fe76e1f85e607814b6a47e
Runner Logs: https://gist.github.com/bkonicek-calm/d0ac3f02187a1975b78425d38c27e918

Additional Context

We are using a custom runner image, but haven't changed it since adding the persistent cache volume, and all we are doing in it is installing a few additional packages.

Start with the gha-runner-scale-set Helm values and EphemeralRunnerSet resource definition in the report, then compare them with TROUBLESHOOTING.md and charts/actions-runner-controller/docs/UPGRADING.md. Reproduce the Go dependency and helmfile cache writes, using the linked controller and runner logs to trace the permission failure. Done means jobs can write to the mounted cache volume without errors.