[Bug]: Pods not starting/creating in K3sContainer

Module

K3S

Testcontainers version

1.20.4

Using the latest Testcontainers version?

Yes

Host OS

Darwin Mac-C6VD4H4XT3 24.1.0 Darwin Kernel Version 24.1.0: Thu Oct 10 21:00:32 PDT 2024; root:xnu-11215.41.3~2/RELEASE_ARM64_T6030 arm64

Host Arch

ARM

Docker version

podman version
Client:       Podman Engine
Version:      5.3.0
API Version:  5.3.0
Go Version:   go1.23.3
Git Commit:   874bf2c301ecf0ba645f1bb45f81966cc755b7da
Built:        Tue Nov 12 17:10:17 2024
OS/Arch:      darwin/arm64

Server:       Podman Engine
Version:      5.2.4
API Version:  5.2.4
Go Version:   go1.22.7
Built:        Mon Oct  7 02:00:00 2024
OS/Arch:      linux/arm64

What happened?

Using K3sContainer in my quarkus-operator tests i got the following situation:
The started k3s Instance is not able to start pods:

k --kubeconfig k3s.yaml get pods -A
NAMESPACE     NAME                                      READY   STATUS              RESTARTS   AGE
kube-system   coredns-5dd589bf46-5d52g                  0/1     ContainerCreating   0          82s
kube-system   local-path-provisioner-846b9dcb6c-wpv92   0/1     ContainerCreating   0          82s
kube-system   metrics-server-5dc58b587c-b24xs           0/1     ContainerCreating   0          82s

Nothing to see in Events:

k --kubeconfig k3s.yaml get events
LAST SEEN   TYPE      REASON                           OBJECT              MESSAGE
2m18s       Normal    Starting                         node/55256e2feaf5   Starting kubelet.
2m18s       Warning   InvalidDiskCapacity              node/55256e2feaf5   invalid capacity 0 on image filesystem
2m17s       Normal    NodeHasSufficientMemory          node/55256e2feaf5   Node 55256e2feaf5 status is now: NodeHasSufficientMemory
2m17s       Normal    NodeHasNoDiskPressure            node/55256e2feaf5   Node 55256e2feaf5 status is now: NodeHasNoDiskPressure
2m17s       Normal    NodeHasSufficientPID             node/55256e2feaf5   Node 55256e2feaf5 status is now: NodeHasSufficientPID
2m17s       Normal    NodeAllocatableEnforced          node/55256e2feaf5   Updated Node Allocatable limit across pods
2m16s       Normal    Starting                         node/55256e2feaf5
2m17s       Normal    NodeReady                        node/55256e2feaf5   Node 55256e2feaf5 status is now: NodeReady
2m15s       Normal    NodePasswordValidationComplete   node/55256e2feaf5   Deferred node password secret validation complete
2m14s       Normal    Synced                           node/55256e2feaf5   Node synced successfully
2m13s       Normal    RegisteredNode                   node/55256e2feaf5   Node 55256e2feaf5 event: Registered Node 55256e2feaf5 in Controller

Log of k3s container see Relevant LogOutput!!

So i copied the K3sContainer and removed the deprecated method call:
https://github.com/testcontainers/testcontainers-java/blob/5628192b6ad620d5dfbcd5b362be5fa227ab531e/modules/k3s/src/main/java/org/testcontainers/k3s/K3sContainer.java#L42
After that the started k3s container was able to create/run pods:

k --kubeconfig k3s.yaml get pods -A
NAMESPACE     NAME                                      READY   STATUS    RESTARTS   AGE
kube-system   coredns-5dd589bf46-922br                  1/1     Running   0          30s
kube-system   local-path-provisioner-846b9dcb6c-kzpxp   1/1     Running   0          30s
kube-system   metrics-server-5dc58b587c-phtjj           1/1     Running   0          30s

I am not sure what the purpose of the deprecated method call is. But after removing it for me everything worked as expected.

Relevant log output

E1206 06:31:12.070270      45 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: unable to apply cgroup configuration: failed to write 4507: write /sys/fs/cgroup/kubepods/besteffort/pod0a22ad1e-6958-416b-8f43-1533890c66eb/8a01e3b65ce62235519eb09b26a0b3bb79e03d6057711e9e0d2e7651c3c9e339/cgroup.procs: no such file or directory: unknown"
E1206 06:31:12.070300      45 kuberuntime_sandbox.go:72] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: unable to apply cgroup configuration: failed to write 4507: write /sys/fs/cgroup/kubepods/besteffort/pod0a22ad1e-6958-416b-8f43-1533890c66eb/8a01e3b65ce62235519eb09b26a0b3bb79e03d6057711e9e0d2e7651c3c9e339/cgroup.procs: no such file or directory: unknown" pod="kube-system/local-path-provisioner-846b9dcb6c-wpv92"
E1206 06:31:12.070310      45 kuberuntime_manager.go:1168] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: unable to apply cgroup configuration: failed to write 4507: write /sys/fs/cgroup/kubepods/besteffort/pod0a22ad1e-6958-416b-8f43-1533890c66eb/8a01e3b65ce62235519eb09b26a0b3bb79e03d6057711e9e0d2e7651c3c9e339/cgroup.procs: no such file or directory: unknown" pod="kube-system/local-path-provisioner-846b9dcb6c-wpv92"
E1206 06:31:12.070337      45 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"local-path-provisioner-846b9dcb6c-wpv92_kube-system(0a22ad1e-6958-416b-8f43-1533890c66eb)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"local-path-provisioner-846b9dcb6c-wpv92_kube-system(0a22ad1e-6958-416b-8f43-1533890c66eb)\\\": rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: unable to apply cgroup configuration: failed to write 4507: write /sys/fs/cgroup/kubepods/besteffort/pod0a22ad1e-6958-416b-8f43-1533890c66eb/8a01e3b65ce62235519eb09b26a0b3bb79e03d6057711e9e0d2e7651c3c9e339/cgroup.procs: no such file or directory: unknown\"" pod="kube-system/local-path-provisioner-846b9dcb6c-wpv92" podUID="0a22ad1e-6958-416b-8f43-1533890c66eb"
W1206 06:31:15.161143      45 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods/besteffort/pod0a22ad1e-6958-416b-8f43-1533890c66eb/8a01e3b65ce62235519eb09b26a0b3bb79e03d6057711e9e0d2e7651c3c9e339 WatchSource:0}: container "8a01e3b65ce62235519eb09b26a0b3bb79e03d6057711e9e0d2e7651c3c9e339" in namespace "k8s.io": not found

Additional Information

No response

Esamina modules/k3s/src/main/java/org/testcontainers/k3s/K3sContainer.java intorno alla riga 42, concentrandoti sulla chiamata al metodo deprecato identificata nel report. Riproduci l'avvio di K3sContainer su Podman ARM e confronta gli errori cgroup e gli stati dei pod prima e dopo la chiamata; il lavoro è concluso quando i pod di sistema raggiungono Running senza i fallimenti della sandbox segnalati.