`service` containers not working on runners with `containerMode: kubernetes`

Controller Version

0.25.2

Helm Chart Version

0.20.2

CertManager Version

1.9.1

Deployment Method

Helm

cert-manager installation

Yes I've followed https://github.com/actions-runner-controller/actions-runner-controller#installation and installed cert-manager from the official source https://cert-manager.io/docs/installation/helm/

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
• I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
• My actions-runner-controller version (v0.x.y) does support the feature
• I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue

Resource Definitions

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerSet
metadata:
  name: k8s-runner
  namespace: actions-runner-system
spec:
  replicas: 4
  organization: devx-ibp
  containerMode: kubernetes
  serviceAccountName: runner-service-account
  selector:
    matchLabels:
      app: k8s-runner
  serviceName: k8s-runner
  template:
    metadata:
      labels:
        app: k8s-runner
  workVolumeClaimTemplate:
    storageClassName: standard
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 1Gi
  labels:
  - k8s-runner
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: runner-role
  namespace: actions-runner-system
rules:
- apiGroups: [ "" ]
  resources: [ "pods" ]
  verbs: [ "get", "list", "create", "delete" ]
- apiGroups: [ "" ]
  resources: [ "pods/exec" ]
  verbs: [ "get", "create" ]
- apiGroups: [ "" ]
  resources: [ "pods/log" ]
  verbs: [ "get", "list", "watch", ]
- apiGroups: [ "batch" ]
  resources: [ "jobs" ]
  verbs: [ "get", "list", "create", "delete" ]
- apiGroups: [ "" ]
  resources: [ "secrets" ]
  verbs: [ "get", "list", "create", "delete" ]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: runner-role-binding
  namespace: actions-runner-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: runner-role
subjects:
- kind: ServiceAccount
  name: runner-service-account
  namespace: actions-runner-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: runner-service-account
  namespace: actions-runner-system


Storage Class:

Name:            standard
IsDefaultClass:  Yes
Annotations:     kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"},"name":"standard"},"provisioner":"rancher.io/local-path","reclaimPolicy":"Delete","volumeBindingMode":"WaitForFirstConsumer"}
,storageclass.kubernetes.io/is-default-class=true
Provisioner:           rancher.io/local-path
Parameters:            <none>
AllowVolumeExpansion:  <unset>
MountOptions:          <none>
ReclaimPolicy:         Delete
VolumeBindingMode:     WaitForFirstConsumer
Events:                <none>

To Reproduce

Execute the following workflow:

name: Go

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  build:
    runs-on: [ self-hosted, k8s-runner ]
    services:
      redis:
        image: redis
        ports:
        - 6379/tcp
    container:
      image: golang:alpine
    steps:
    - uses: actions/checkout@v3
    - run: go build cmd/hello/main.go
    - run: ./main

Describe the bug

The initialize container step fails:

5s
##[debug]Evaluating condition for step: 'Initialize containers'
##[debug]Evaluating: success()
##[debug]Evaluating success:
##[debug]=> true
##[debug]Result: true
##[debug]Starting: Initialize containers
##[debug]Register post job cleanup for stopping/deleting containers.
Run '/runner/k8s/index.js'
##[debug]/runner/externals/node[1](https://github.com/devx-ibp/bquenin-actions/runs/8142952627?check_suite_focus=true#step:3:1)[6](https://github.com/devx-ibp/bquenin-actions/runs/8142952627?check_suite_focus=true#step:3:6)/bin/node /runner/k[8](https://github.com/devx-ibp/bquenin-actions/runs/8142952627?check_suite_focus=true#step:3:8)s/index.js
##[debug]Using image 'golang:alpine' for job image
##[debug]Adding service 'redis' to pod definition
Error: Error: failed to create job pod: HttpError: HTTP request failed
Error: Process completed with exit code 1.
Error: Executing the custom container implementation failed. Please contact your self hosted runner administrator.
##[debug]System.Exception: Executing the custom container implementation failed. Please contact your self hosted runner administrator.
##[debug] ---> System.Exception: The hook script at '/runner/k8s/index.js' running command 'PrepareJob' did not execute successfully
##[debug]   at GitHub.Runner.Worker.Container.ContainerHooks.ContainerHookManager.ExecuteHookScript[T](IExecutionContext context, HookInput input, ActionRunStage stage, String prependPath)
##[debug]   --- End of inner exception stack trace ---
##[debug]   at GitHub.Runner.Worker.Container.ContainerHooks.ContainerHookManager.ExecuteHookScript[T](IExecutionContext context, HookInput input, ActionRunStage stage, String prependPath)
##[debug]   at GitHub.Runner.Worker.Container.ContainerHooks.ContainerHookManager.PrepareJobAsync(IExecutionContext context, List`1 containers)
##[debug]   at GitHub.Runner.Worker.ContainerOperationProvider.StartContainersAsync(IExecutionContext executionContext, Object data)
##[debug]   at GitHub.Runner.Worker.JobExtensionRunner.RunAsync()
##[debug]   at GitHub.Runner.Worker.StepsRunner.RunStepAsync(IStep step, CancellationToken jobCancellationToken)
##[debug]Finishing: Initialize containers

Describe the expected behavior

Hi,

I'm trying to use a service container in a job. I was expecting the service container to be created as an additional container to the pod executing this job but it looks like it's not working. Is there anything I'm missing?

Controller Logs

https://gist.github.com/bquenin/ddbe50c71dadd6b136ab0b0b5bee6e63

Runner Pod Logs

https://gist.github.com/bquenin/ddbe50c71dadd6b136ab0b0b5bee6e63

Reproduce the workflow with the Kubernetes container mode, job container, and Redis service, then start at the /runner/k8s/index.js hook and its PrepareJob path. Trace the failed job-pod creation and use the controller and runner pod logs linked in the issue; done means the workflow initializes successfully with the Redis service available.