Ability to run container as arbitrary UID for running on OpenShift

Who is this for and what problem do they have today?

NOTE: I waffled a bit on whether to file as a bug or a feature, but it doesn't look like OpenShift is currently a target environment, so I'm using the Feature request.

I would like to be able to use redpanda as part of my quarkus dev services which I develop and test inside of an OpenShift cluster.

For security purposes, OpenShift by default will run containers by feeding an arbitrary UID. This makes it so that multiple containers running on the same node in a kubernetes cluster don't accidentally run in the same kernel/selinux namespace, which could give them access to eachother's resources on the host.

This is mostly equivalent to running as such:

$ podman run -u 10001 -it docker.redpanda.com/redpandadata/redpanda:v23.2.3

However, when I test the image this way, I start to run into file ownership issues. Here are a few examples from my local testing from the image:

$ podman run -u 10001 --entrypoint bash -it docker.redpanda.com/redpandadata/redpanda:v23.2.3
10001@318ddfa476b3:/$ rpk redpanda start --overprovisioned
WARNING: This is a setup for development purposes only; in this mode your clusters may run unrealistically fast and data can be corrupted any time your computer shuts down uncleanly.
Error: error writing to temporary file: open /etc/redpanda/redpanda-3966681898729425324: permission denied


10001@318ddfa476b3:/$ rpk redpanda start --overprovisioned --check false
WARNING: This is a setup for development purposes only; in this mode your clusters may run unrealistically fast and data can be corrupted any time your computer shuts down uncleanly.
Detecting the current cloud vendor and VM
System check - STARTED
Error: unable to check if system meets redpanda requirements: fatal error during checker "Data directory is writable" execution: open /var/lib/redpanda/data/test_file: permission denied; to override this check you can use --check=false

You'll notice a very similar result if I try to run in an OpenShift cluster as a non cluster-admin user:

$ oc whoami
esauer

$ oc run redpanda --image=docker.redpanda.com/redpandadata/redpanda:v23.2.3 --wait
pod/redpanda created

$ oc logs -f redpanda
+ /usr/local/bin/supervisord -d
+ '[' '' = true ']'
+ exec /usr/bin/rpk redpanda start --overprovisioned
WARNING: This is a setup for development purposes only; in this mode your clusters may run unrealistically fast and data can be corrupted any time your computer shuts down uncleanly.
Error: error writing to temporary file: open /etc/redpanda/redpanda-5315711954022061005: permission denied

What are the success criteria?

I would like to be able to run the redpanda image as a non-privileged user in OpenShift. The above commands should result in running redpanda instances.

Why is solving this problem impactful?

Currently, we have development teams building apps in quarkus, and we have to work around the fact that quarkus dev services/testcontainers doesn't run properly inside of our OpenShift environment. This requires us to do additional setup to test our Kafka consumers/producers using either full on kafka clusters that we set up separately, or using the smallrye in-memory kafka options. Having redpanda (which is the default option for Quarkus Dev Services) work in OpenShift would save us a large chunk of configuration time, as well as time helping other developers troubleshoot why their builds aren't working.

Additional notes

I did try to run some experiments to see if simply modifying file permissions in the docker image to be group owned would help resolve this issue, as that has worked for me with other projects. Example dockerfile:

FROM docker.redpanda.com/redpandadata/redpanda:v23.2.3

USER 0

RUN chgrp -R 0 /var/lib/redpanda && \
    chmod -R g=u /var/lib/redpanda && \
    chgrp -R 0 /etc/redpanda && \
    chmod -R g=u /etc/redpanda

USER 101

Unfortunately, it looks like the go code is actually trying to do some chmoding of its own as the process is running, and this wasn't enough to allow it to work.

JIRA Link: CORE-1400

Start by reproducing the podman -u 10001 command and inspect the image startup path around /usr/local/bin/supervisord and the rpk redpanda start entry point. Trace the reported permission failures under /etc/redpanda and /var/lib/redpanda, including the runtime chmod behavior. Done means the provided commands run successfully as an arbitrary non-privileged UID in OpenShift.