Sysbox installation on Rancher Kubernetes (RKE2) fails

Hi Guys,

I discovered the same problem as in issue 380

It's a RKE2 Cluster with Kubernetes Version 1.31.7 (but also tried with Version 1.31.1 and 1.28.15)
OS are Ubuntu 20.04

The cluster and rancher are installed as described in this document.

After enter kubectl label nodes node2 sysbox-install=yes and kubectl apply -f https://raw.githubusercontent.com/nestybox/sysbox/master/sysbox-k8s-manifests/sysbox-install.yaml the sysbox-deploy-k8s-d956r pod are in a crashloopbackoff with the error:

Detected Kubernetes version v1.28
Adding K8s taint "sysbox-runtime=not-running:NoSchedule" to node ...
node/node2 modified
Deploying Kubelet config agent on the host ...
Running Kubelet config agent on the host (will restart Kubelet and temporary bring down all pods on this node for ~1 min) ...
Job for kubelet-config-helper.service failed because the control process exited with error code. See "systemctl status kubelet-config-helper.service" and "journalctl -xe" for details.

root@node2:~# systemctl status kubelet-config-helper:

● kubelet-config-helper.service - Kubelet config service
Loaded: loaded (/lib/systemd/system/kubelet-config-helper.service; static; vendor preset: enabled)
Active: failed (Result: exit-code) since Wed 2025-04-09 12:22:25 UTC; 3min 1s ago
Process: 43182 ExecStart=/bin/sh -c /usr/local/bin/kubelet-config-helper.sh (code=exited, status=5)
Main PID: 43182 (code=exited, status=5)

Apr 09 12:22:25 node2 sh[43183]: SUB
Apr 09 12:22:25 node2 sh[43183]: 1 ...
Apr 09 12:22:25 node2 sh[43183]: + systemctl stop LOAD ACTIVE SUB 1
Apr 09 12:22:25 node2 sh[43296]: Failed to stop LOAD.service: Unit LOAD.service not loaded.
Apr 09 12:22:25 node2 sh[43296]: Failed to stop ACTIVE.service: Unit ACTIVE.service not loaded.
Apr 09 12:22:25 node2 sh[43296]: Failed to stop SUB.service: Unit SUB.service not loaded.
Apr 09 12:22:25 node2 sh[43296]: Failed to stop 1.service: Unit 1.service not loaded.
Apr 09 12:22:25 node2 systemd[1]: kubelet-config-helper.service: Main process exited, code=exited, status=5/NOTINSTALLED
Apr 09 12:22:25 node2 systemd[1]: kubelet-config-helper.service: Failed with result 'exit-code'.
Apr 09 12:22:25 node2 systemd[1]: Failed to start Kubelet config service.

journalctl -xe:

root@node2:~# journalctl -xe

Subject: Unit process exited
Defined-By: systemd
Support: http://www.ubuntu.com/support

An ExecStart= process belonging to unit kubelet-config-helper.service has exited.

The process' exit code is 'exited' and its exit status is 5.
Apr 09 12:11:21 node2 systemd[1]: kubelet-config-helper.service: Failed with result 'exit-code'.
Subject: Unit failed
Defined-By: systemd
Support: http://www.ubuntu.com/support

The unit kubelet-config-helper.service has entered the 'failed' state with result 'exit-code'.
Apr 09 12:11:21 node2 systemd[1]: Failed to start Kubelet config service.
Subject: A start job for unit kubelet-config-helper.service has failed
Defined-By: systemd
Support: http://www.ubuntu.com/support

A start job for unit kubelet-config-helper.service has finished with a failure.

The job identifier is 5182 and the job result is failed.
Apr 09 12:11:21 node2 systemd[1]: cri-containerd-b969eba95345185a22cfe13920873e26d3b1a665db4d1c02a9d3047522852381.scope: Succeeded.
Subject: Unit succeeded
Defined-By: systemd
Support: http://www.ubuntu.com/support

The unit cri-containerd-b969eba95345185a22cfe13920873e26d3b1a665db4d1c02a9d3047522852381.scope has successfully entered the 'dead' state.
Apr 09 12:11:21 node2 systemd[1]: run-k3s-containerd-io.containerd.runtime.v2.task-k8s.io-b969eba95345185a22cfe13920873e26d3b1a665db4d1c02a9d3047522852381-rootfs.mount: Succeeded.
Subject: Unit succeeded
Defined-By: systemd
Support: http://www.ubuntu.com/support

The unit run-k3s-containerd-io.containerd.runtime.v2.task-k8s.io-b969eba95345185a22cfe13920873e26d3b1a665db4d1c02a9d3047522852381-rootfs.mount has successfully entered the 'dead' state.
Apr 09 12:11:24 node2 snapd[36332]: daemon.go:548: gracefully waiting for running hooks
Apr 09 12:11:24 node2 snapd[36332]: daemon.go:550: done waiting for running hooks
Apr 09 12:11:27 node2 snapd[36332]: overlord.go:518: Released state lock file
Apr 09 12:11:27 node2 snapd[36332]: daemon stop requested to wait for socket activation
Apr 09 12:11:27 node2 systemd[1]: snapd.service: Succeeded.
Subject: Unit succeeded
Defined-By: systemd
Support: http://www.ubuntu.com/support

The unit snapd.service has successfully entered the 'dead' state.`

I've tried to figure out if I need special requirements for RKE2 but as described in https://github.com/nestybox/sysbox/blob/master/docs/user-guide/install-k8s.md#kubernetes-worker-node-requirements it should work out of the box.

Am I missing something?

This is a test-cluster. So if anybody needs access to it let me know.

Thanks in advance and best regards
Franz