Initialize Containers - HttpError: HTTP request failed - EKS - containerMode kubernetes

When I attempt to run a workflow against a self-hosted runner deployed using the gha-runner-scale-set-controller and gha-runner-scale-set charts, my job fails on the 'Initialize Containers' step.

Runner Scale Set values.yaml:

minRunners: 1
maxRunners: 16

containerMode:
  type: kubernetes
  kubernetesModeWorkVolumeClaim:
    accessModes: ["ReadWriteOnce"]
    storageClassName: "ebs-gp3-ephemeral"
    resources:
      requests:
        storage: 10Gi

template:
  spec:
    securityContext:
      fsGroup: 123
    containers:
    - name: runner
      image: ghcr.io/actions/actions-runner:latest
      command: ["/home/runner/run.sh"]

In the Github UI after the job is picked up the following error message appears in the log:

Error: HttpError: HTTP request failed
Error: Process completed with exit code 1.
Error: Executing the custom container implementation failed. Please contact your self hosted runner administrator.

Full error context:

##[debug]Evaluating condition for step: 'Initialize containers'
##[debug]Evaluating: success()
##[debug]Evaluating success:
##[debug]=> true
##[debug]Result: true
##[debug]Starting: Initialize containers
##[debug]Register post job cleanup for stopping/deleting containers.
Run '/home/runner/k8s/index.js'
##[debug]/home/runner/externals/node1[6](https://github.com/reverbdotcom/reverb-terraform/actions/runs/7426318988/job/20311530393#step:2:6)/bin/node /home/runner/k[8](https://github.com/reverbdotcom/reverb-terraform/actions/runs/7426318988/job/20311530393#step:2:8)s/index.js
Error: HttpError: HTTP request failed
Error: Process completed with exit code 1.
Error: Executing the custom container implementation failed. Please contact your self hosted runner administrator.
##[debug]System.Exception: Executing the custom container implementation failed. Please contact your self hosted runner administrator.
##[debug] ---> System.Exception: The hook script at '/home/runner/k8s/index.js' running command 'PrepareJob' did not execute successfully
##[debug]   at GitHub.Runner.Worker.Container.ContainerHooks.ContainerHookManager.ExecuteHookScript[T](IExecutionContext context, HookInput input, ActionRunStage stage, String prependPath)
##[debug]   --- End of inner exception stack trace ---
##[debug]   at GitHub.Runner.Worker.Container.ContainerHooks.ContainerHookManager.ExecuteHookScript[T](IExecutionContext context, HookInput input, ActionRunStage stage, String prependPath)
##[debug]   at GitHub.Runner.Worker.Container.ContainerHooks.ContainerHookManager.PrepareJobAsync(IExecutionContext context, List`1 containers)
##[debug]   at GitHub.Runner.Worker.ContainerOperationProvider.StartContainersAsync(IExecutionContext executionContext, Object data)
##[debug]   at GitHub.Runner.Worker.JobExtensionRunner.RunAsync()
##[debug]   at GitHub.Runner.Worker.StepsRunner.RunStepAsync(IStep step, CancellationToken jobCancellationToken)
##[debug]Finishing: Initialize containers

My workflow:

name: git hooks
on: push

jobs:
  pre-commit:
    name: pre-commit
    runs-on: reverbdotcom-general-purpose
    container: summerwind/actions-runner:latest
    steps:
      - run: echo "hello actions"

I have tried a lot of different things to try to understand what is not working here but the chain of dependencies and effects is not easy to comprehend. There is a lot of red herrings and other noise in the logs which led me on several chases around the web, and I spent a while trying security contexts, various container images, etc. At this point I think I have run out of time to figure this out and will have to fall back to the previous actions runner controller and advise my team that the next generation of actions runners is a risk and we should evaluate alternative CI pipelines.

Start with the mentioned /home/runner/k8s/index.js entry point and the PrepareJob hook in the supplied debug output. Reproduce the failure with the provided Runner Scale Set values and workflow, then trace the HTTP request behind Initialize Containers; done means the container hook completes and the job proceeds past that step.