Epic: Run k3s natively in Gitpod

Right now, to run k3s in Gitpod the only viable option is to use emulation to create a VM as showed here https://github.com/gitpod-io/template-k3s

I did some analysis of why, even if we are able to run Docker, k3s still does not work.

There are a couple of errors to solve

Error 1 Snapshotter (✔️ )

ERRO[2021-07-21T08:57:21.159608731Z] Failed to configure agent: "overlayfs" snapshotter cannot be enabled for "/var/lib/rancher/k3s/agent/containerd", try using "fuse-overlayfs" or "native": failed to mount overlay: invalid argument

Solution:

This is already solved, we can use --snapshotter=fuse-overlayfs since we fixed fuse support in https://github.com/gitpod-io/gitpod/pull/4594 and https://github.com/gitpod-io/gitpod/pull/4762

Error 2 Privileges (❎ )

E0721 09:00:38.387724   20938 server.go:288] "Failed to run kubelet" err="failed to run Kubelet: failed to create kubelet: open /dev/kmsg: no such file or directory"

The kubelet can run in rootless mode to avoid us dealing with privileged devices, like we do for Docker.

However, running in that mode

 sudo k3s server --snapshotter=fuse-overlayfs --rootless
WARN[2021-07-21T09:01:57.964650205Z] cannot set cgroup2 evacuation, make sure to run k3s as a systemd unit 
WARN[2021-07-21T09:01:57.965501805Z] Running RootlessKit as the root user is unsupported. 
WARN[2021-07-21T09:01:57.965778310Z] The host root filesystem is mounted as "". Setting child propagation to "" is not supported. 
FATA[2021-07-21T09:01:57.969073870Z] failed to setup UID/GID map: failed to compute uid/gid map: No subuid ranges found for user 0 ("root") 

Amazing, we can create the uid and gid maps for that user by editing /etc/subuid and /etc/subgid

However, after doing that

sudo k3s server --snapshotter=fuse-overlayfs --rootless
WARN[2021-07-21T09:09:54.117789745Z] cannot set cgroup2 evacuation, make sure to run k3s as a systemd unit 
WARN[2021-07-21T09:09:54.119755834Z] Running RootlessKit as the root user is unsupported. 
WARN[2021-07-21T09:09:54.120024215Z] The host root filesystem is mounted as "". Setting child propagation to "" is not supported. 
FATA[2021-07-21T09:09:54.125484467Z] failed to setup UID/GID map: newuidmap 24262 [0 0 1 1 200000 65536] failed: newuidmap: write to uid_map failed: Operation not permitted
: exit status 1 

Looks like the problem is that the current workspace root user cannot write the uid map of any process.

24548 openat(AT_FDCWD, "/proc/24262/", O_RDONLY|O_DIRECTORY) = 3
24548 openat(3, "uid_map", O_WRONLY)    = -1 EACCES (Permission denied)
24548 write(2, "newuidmap: open of uid_map faile"..., 53) = 53

Front conversations

Start by reproducing the documented k3s server --snapshotter=fuse-overlayfs --rootless commands and reviewing /etc/subuid, /etc/subgid, and the reported /proc/.../uid_map failure. Done means k3s runs natively in Gitpod without VM emulation and the privilege error is resolved; the issue does not identify implementation files or tests.