S2i injection feature is broken on stock Ubuntu 20.04 LTS systems

Is this a feature request or bug?

/kind bug

What went wrong?

On Ubuntu 20.04 system, I cannot build a docker image which uses the injection feature.
It used to work on Ubuntu < 20.04 and still works on CentOS 7.x systems.

This bug has already been reported as #1006 but was closed for lifecycle/rotten.
This bug report should have all the information to reproduce, workaround and fix the issue.

Steps to reproduce:

Run this on fresh Ubuntu 20.04 system with docker installed (sudo apt-get install docker.io).
Attention: do not tweak the default value of the fs.protected_regular sysctl, which should be 1.

1. docker pull quay.io/centos7/ruby-27-centos7
2. echo hello > file.txt
3. s2i build https://github.com/openshift/ruby-hello-world.git quay.io/centos7/ruby-27-centos7 my-hello-world:xxx --inject $PWD/file.txt:/bug.txt (fails)
4. echo $? (displays 1)

Expected results:

I expect the image to build successfully.

Actual results:

The image fails to build, with the following error message at the end:

---> Cleaning up unused ruby gems ...
Running `bundle clean --verbose` with bundler 2.1.4
Frozen, using resolution from the lockfile
truncate: cannot open '/tmp/rm-injections' for writing: Permission denied
Build failed
ERROR: An error occurred: non-zero (13) exit code from quay.io/centos7/ruby-27-centos7

Version:

s2i: s2i v1.3.1
docker:

Client:
 Version:           20.10.2
 API version:       1.41
 Go version:        go1.13.8
 Git commit:        20.10.2-0ubuntu1~20.04.2
 Built:             Tue Mar 30 21:24:57 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.2
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.8
  Git commit:       20.10.2-0ubuntu1~20.04.2
  Built:            Mon Mar 29 19:10:09 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.3.3-0ubuntu2.3
  GitCommit:        
 runc:
  Version:          spec: 1.0.2-dev
  GitCommit:        
 docker-init:
  Version:          0.19.0
  GitCommit:        

Additional info:

The bug happens on stock Ubuntu 20.04 LTS system.
It does not happen on Centos 7.x or on Ubuntu 14.04 systems that I have access to.

The reason is that on Ubuntu 20.04 LTS sudo sysctl fs.protected_regular=1 is the default,
which (according to https://unix.stackexchange.com/questions/503111/group-permissions-for-root-not-working-in-tmp ) restricts the permissions under /tmp and in particular prevents s2i from truncating the /tmp/rm-injections file inside the container despite that file being writable by everyone (chmod 0666).

At the end of the s2i assemble phase, s2i will try to truncate the /tmp/rm-injections file, which has mode 0666 (writable by everyone). This file is owned by user 1000:1000 while the s2i assemble user has uid 1001:1001. The assemble user is then denied the right to truncate the /tmp/rm-injections file, despite that file being world-writable.

Here is to reproduce the problem with a simple busybox container:

sudo sysctl fs.protected_regular=1
docker run -d --name=permbug -u 1001:1001 --entrypoint=/bin/sh busybox:latest -c 'sleep 10000'
docker exec -u 0 permbug /bin/sh -c 'echo blah > /tmp/xxx ; chown 1000:1000 /tmp/xxx ; chmod 666 /tmp/xxx'
docker exec -u 1001:1001 permbug /bin/sh -c 'truncate -s0 /tmp/xxx && echo "WORKS" || echo "BUG"'
docker rm -f permbug

We see that if the run the same command creating the xxx file in /var instead of /tmp it works:

sudo sysctl fs.protected_regular=1
docker run -d --name=permbug -u 1001:1001 --entrypoint=/bin/sh busybox:latest -c 'sleep 10000'
docker exec -u 0 permbug /bin/sh -c 'echo blah > /var/xxx ; chown 1000:1000 /var/xxx ; chmod 666 /var/xxx'
docker exec -u 1001:1001 permbug /bin/sh -c 'truncate -s0 /var/xxx && echo "WORKS" || echo "BUG"'
docker rm -f permbug

If we force sudo sysctl fs.protected_regular=0 in the first example it works again:

sudo sysctl fs.protected_regular=0
docker run -d --name=permbug -u 1001:1001 --entrypoint=/bin/sh busybox:latest -c 'sleep 10000'
docker exec -u 0 permbug /bin/sh -c 'echo blah > /tmp/xxx ; chown 1000:1000 /tmp/xxx ; chmod 666 /tmp/xxx'
docker exec -u 1001:1001 permbug /bin/sh -c 'truncate -s0 /tmp/xxx && echo "WORKS" || echo "BUG"'
docker rm -f permbug

To workaround this issue I had to:

• sudo sysctl fs.protected_regular=0 (in a /etc/sysctl.d file) (best workaround)
• hack the s2i binary with sed -i -e s:tmp/rm-injections:var/rm-injections:g (giant hack)

To fix the issue properly, I suggest either:

• add a command-line option to allow creating the /tmp/rm-injections file outside of /tmp
• hardcode the value of rmInjectionsScript in https://github.com/openshift/source-to-image/blob/30d81a9440f30b472bb32e592b12c1a83a396edd/pkg/build/strategies/sti/sti.go#L35 to a path that is not tmp-like, for example to /var/rm-injections.
• add a FAQ entry in the project README.md advising to create a /etc/sysctl.d entry with fs.protected_regular=0 on Ubuntu 20.04 and more recent systems.

Start with pkg/build/strategies/sti/sti.go at the rmInjectionsScript reference and reproduce the failure on a stock Ubuntu 20.04 system using the commands in the issue. Compare the injection behavior under /tmp and /var, then run the relevant build tests if available. Done means the injection build succeeds with fs.protected_regular=1 without requiring a host sysctl workaround.