[CONNECT] run as non-root: launcher service account not found

Our kubernetes environment only permits non-root, read-only containers and I was experimenting to see if I could get connect with launcher to run successfully like this. It looks like I succeeded but the launcher is complaining about not finding the default service account in the job namespace (it definitely exists) - can you shed any light on what it needs? Is there some part of the setup that I am missing? I am very excited about getting the off-host execution working

Also, it would be great if the non-root setup could be incorporated into the helm chart! I had to bypass the prestart script (which tries to run update-ca-certificates and requires being root), but it could just read the root ca from the filesystem and set the KUBERNETES_CA_CERT_DATA environment variable. Even better, the cert is already on the file system so maybe it could be picked up from there rather than having to be specified.

The helm chart values file looks like this:

sharedStorage:
  mount: true
  name: rsc-pvc

prometheusExporter:
  enabled: false

launcher:
  enabled: true
  namespace: rsc-job

args:
  - /usr/local/bin/startup.sh

config:

  Database:
    Provider: "Postgres"
  Postgres:
    URL: "postgres://<host>/connectdb"

  Launcher:
    DataDirPVCName: rsc-pvc
    KubernetesCACertificateData: <base-64-encoded-cert-data>

  Metrics:
    Enabled: false

pod:
  volumeMounts:
    - name: tmp
      mountPath: "/tmp"
    - name: bind-mount
      mountPath: "/opt/rstudio-connect/mnt"
    - name: launcher-conf
      mountPath: "/etc/rstudio-connect/launcher"
    - name: launcher-scratch
      mountPath: "/var/lib/rstudio-connect-launcher"
  volumes:
    - name: tmp
      emptyDir:
        sizeLimit: 1Gi
    - name: bind-mount
      emptyDir:
        sizeLimit: 1Gi
    - name: launcher-conf
      emptyDir:
        sizeLimit: 1Gi
    - name: launcher-scratch
      emptyDir:
        sizeLimit: 1Gi
  securityContext:
    seccompProfile:
      type: RuntimeDefault

securityContext:
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  privileged: false
  runAsNonRoot: true
  capabilities:
    drop:
    - ALL  
  runAsUser: 999
  runAsGroup: 999

Console log runs through the config tasks fine and ends like this:

time="2023-06-01T19:59:39.852Z" level=info msg="Configuration tasks: complete"
time="2023-06-01T20:00:12.988Z" level=info msg="Using the normalized Server.Address: http://localhost:3939/"
time="2023-06-01T20:00:13.366Z" level=info msg="Waiting until launcher is started..."
time="2023-06-01T20:01:12.366Z" level=info msg="Starting the launcher..."
time="2023-06-01T20:01:12.377Z" level=info msg="2023-06-01T20:01:12.371167Z [rstudio-launcher] ERROR system error 1 (Operation not permitted) [path: /var/lib/rstudio-connect-launcher, description: Could not change permissions on path /var/lib/rstudio-connect-launcher. Is root squash enabled?]; OCCURRED AT rstudio::core::Error rstudio::core::{anonymous}::changeFileModeImpl(const string&, mode_t) src/cpp/shared_core/FilePath.cpp:317; LOGGED FROM: rstudio::core::Error rstudio::job_launcher::Config::impl::validate() src/cpp/job_launcher/LauncherConfig.cpp:406" stream=stderr subprocess=rstudio-launcher
time="2023-06-01T20:01:12.377Z" level=info msg="2023-06-01T20:01:12.371299Z [rstudio-launcher] INFO Running in unprivileged single-user mode" stream=stderr subprocess=rstudio-launcher
time="2023-06-01T20:01:12.377Z" level=info msg="2023-06-01T20:01:12.372208Z [rstudio-launcher] INFO Bootstrapping plugin Kubernetes" stream=stderr subprocess=rstudio-launcher
time="2023-06-01T20:01:12.384Z" level=info msg="2023-06-01T20:01:12.384261Z [rstudio-kubernetes-launcher, log-source: Kubernetes] WARNING Could not set permissions on scratch path (/var/lib/rstudio-connect-launcher/Kubernetes)- it is recommended to set them to rwxr-xr-x; LOGGED FROM: rstudio::core::ProgramStatus rstudio::job_launcher::impls::entry_point::initialize(int, char**, const string&, rstudio::job_launcher::impls::OptionsBase*, rstudio_boost::shared_ptr<rstudio::job_launcher::impls::FrameworkCommunicator>&) src/cpp/job_launcher/impls/EntryPoint.cpp:207" stream=stderr subprocess=rstudio-launcher
time="2023-06-01T20:01:12.431Z" level=info msg="2023-06-01T20:01:12.431274Z [rstudio-kubernetes-launcher, log-source: Kubernetes] INFO Permission to list nodes denied by the Kubernetes API server. This is fine in most cases, but IP addresses for jobs with a NodePort will reflect only the cluster's own IP range -- which may not be visible to an external service. Consider adding a nodes' ClusterRole for the ServiceAccount if network connectivity from outside the cluster is an issue." stream=stderr subprocess=rstudio-launcher
time="2023-06-01T20:01:12.488Z" level=info msg="2023-06-01T20:01:12.482557Z [rstudio-kubernetes-launcher, log-source: Kubernetes] INFO Pruning 0 jobs" stream=stderr subprocess=rstudio-launcher
time="2023-06-01T20:01:12.493Z" level=info msg="2023-06-01T20:01:12.492615Z [rstudio-launcher] INFO Initializing SslAsyncServer [address: 0.0.0.0, port: 5559]" stream=stderr subprocess=rstudio-launcher
time="2023-06-01T20:01:12.504Z" level=info msg="2023-06-01T20:01:12.504260Z [rstudio-launcher] INFO Running server" stream=stderr subprocess=rstudio-launcher
time="2023-06-01T20:01:12.515Z" level=info msg="Launcher started."
time="2023-06-01T20:01:12.522Z" level=fatal msg="Error: Cannot setup launcher service accounts: Launcher is unable to find configured global default service account: default in namespace: rsc-job"

Start by reviewing the Helm chart values and the prestart script, then reproduce the launcher startup with the provided non-root security context and logs. Investigate the configured service account in the rsc-job namespace and the read-only filesystem requirements. Done should include a documented or chart-supported non-root setup that starts the launcher and resolves service-account configuration.