Runner doesn't set proper permissions for mounted folders in container jobs with rootless docker images

Hello everyone,
I am creating this issue based on user's report that we got in https://github.com/actions/runner-images/issues/6930

Describe the bug
When user runs container job with root-less docker image (use custom user instead of root), this user won't have write access to any mounted folders and job will fail on actions/checkout step or echo "test" > $GITHUB_ENV

To Reproduce

1. Build root-less docker image:

   FROM "ubuntu:20.04"
   RUN useradd github
   USER github
   

2. Run simple workflow:

   runs-on: ubuntu-20.04 # Standard Runner
   container:
         image: <my_rootless_image>
   steps:
    - run: |
        echo "test=123" > $GITHUB_ENV
    - uses: actions/checkout@v3
   

3. Look at the results:
   Step echo "test=123" > $GITHUB_ENV will fail with

If we remove the first step and just run actions/checkout, it will fail with

Expected behavior
Root-less docker image should work as expected.

Runner Version and Platform

Latest Agent Version, Linux platform

Possible root cause

If you invoke ls -la $GITHUB_ENV and ls -la $HOME, you will notice that both files / folders are owned by user with UID 1001

$ ls -la $GITHUB_ENV
-rw-r--r-- 1 1001 122 0 Feb  1 11:41 /__w/_temp/_runner_file_commands/set_env_27af1008-076d-41ad-8cf9-a0eb8e02f3e4

$ ls -la $HOME
total 8
drwxr-xr-x 2 1001  122 4096 Feb  1 11:41 .
drwxr-xr-x 4 root root 4096 Feb  1 11:41 ..

Then if you invoke cat /etc/passwd, you will notice that user with UID 1001 doesn't exist in docker container at all:

$ cat /etc/passwd
...
github:x:1000:1000::/home/github:/bin/sh

But user with UID 1001 exists on host machine (it is runner user):
So looks like when you mount folders from host machine and those folders are owned by user runner with UID 1001 on host machine, they belongs to user with UID 1001 inside container too. Despite the fact that this user doesn't exist at all.
So these folders are not available for github user in container which has UID 1000.

One possible workaround is creating new user during docker build with UID 1001 to make sure that this UID is matched to UID of runner user on standard runners:

FROM "ubuntu:20.04"
# Add a user with id 1001 to match permissions with the github env.
RUN useradd -u 1001 -M github
USER 1001

But this workaround doesn't work for Larger Runners because on Larger Runners runner user has UID 1000 instead of 1001 on Standard Runners.
And I guess it will be a problem for some self-hosted agents too where user UID can be random.

Should agent set proper permissions for all mounted folders to make sure that they are available for docker container user regardless of its UID?

Start by reproducing the supplied workflow with a rootless Docker image on a Linux runner, then inspect the runner's container-job mount and permission handling. The change is complete when the non-root container user can write to mounted folders, including GITHUB_ENV, and actions/checkout succeeds across differing runner UIDs.