[Bug]: Pods not starting/creating in K3sContainer

Module

K3S

Testcontainers version

1.20.4

Using the latest Testcontainers version?

Yes

Host OS

Darwin Mac-C6VD4H4XT3 24.1.0 Darwin Kernel Version 24.1.0: Thu Oct 10 21:00:32 PDT 2024; root:xnu-11215.41.3~2/RELEASE_ARM64_T6030 arm64

Host Arch

ARM

Docker version

podman version
Client:       Podman Engine
Version:      5.3.0
API Version:  5.3.0
Go Version:   go1.23.3
Git Commit:   874bf2c301ecf0ba645f1bb45f81966cc755b7da
Built:        Tue Nov 12 17:10:17 2024
OS/Arch:      darwin/arm64

Server:       Podman Engine
Version:      5.2.4
API Version:  5.2.4
Go Version:   go1.22.7
Built:        Mon Oct  7 02:00:00 2024
OS/Arch:      linux/arm64

What happened?

Using K3sContainer in my quarkus-operator tests i got the following situation:
The started k3s Instance is not able to start pods:

k --kubeconfig k3s.yaml get pods -A
NAMESPACE     NAME                                      READY   STATUS              RESTARTS   AGE
kube-system   coredns-5dd589bf46-5d52g                  0/1     ContainerCreating   0          82s
kube-system   local-path-provisioner-846b9dcb6c-wpv92   0/1     ContainerCreating   0          82s
kube-system   metrics-server-5dc58b587c-b24xs           0/1     ContainerCreating   0          82s

Nothing to see in Events:

k --kubeconfig k3s.yaml get events
LAST SEEN   TYPE      REASON                           OBJECT              MESSAGE
2m18s       Normal    Starting                         node/55256e2feaf5   Starting kubelet.
2m18s       Warning   InvalidDiskCapacity              node/55256e2feaf5   invalid capacity 0 on image filesystem
2m17s       Normal    NodeHasSufficientMemory          node/55256e2feaf5   Node 55256e2feaf5 status is now: NodeHasSufficientMemory
2m17s       Normal    NodeHasNoDiskPressure            node/55256e2feaf5   Node 55256e2feaf5 status is now: NodeHasNoDiskPressure
2m17s       Normal    NodeHasSufficientPID             node/55256e2feaf5   Node 55256e2feaf5 status is now: NodeHasSufficientPID
2m17s       Normal    NodeAllocatableEnforced          node/55256e2feaf5   Updated Node Allocatable limit across pods
2m16s       Normal    Starting                         node/55256e2feaf5
2m17s       Normal    NodeReady                        node/55256e2feaf5   Node 55256e2feaf5 status is now: NodeReady
2m15s       Normal    NodePasswordValidationComplete   node/55256e2feaf5   Deferred node password secret validation complete
2m14s       Normal    Synced                           node/55256e2feaf5   Node synced successfully
2m13s       Normal    RegisteredNode                   node/55256e2feaf5   Node 55256e2feaf5 event: Registered Node 55256e2feaf5 in Controller

Log of k3s container see Relevant LogOutput!!

So i copied the K3sContainer and removed the deprecated method call:
https://github.com/testcontainers/testcontainers-java/blob/5628192b6ad620d5dfbcd5b362be5fa227ab531e/modules/k3s/src/main/java/org/testcontainers/k3s/K3sContainer.java#L42
After that the started k3s container was able to create/run pods:

k --kubeconfig k3s.yaml get pods -A
NAMESPACE     NAME                                      READY   STATUS    RESTARTS   AGE
kube-system   coredns-5dd589bf46-922br                  1/1     Running   0          30s
kube-system   local-path-provisioner-846b9dcb6c-kzpxp   1/1     Running   0          30s
kube-system   metrics-server-5dc58b587c-phtjj           1/1     Running   0          30s

I am not sure what the purpose of the deprecated method call is. But after removing it for me everything worked as expected.

Relevant log output

E1206 06:31:12.070270      45 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: unable to apply cgroup configuration: failed to write 4507: write /sys/fs/cgroup/kubepods/besteffort/pod0a22ad1e-6958-416b-8f43-1533890c66eb/8a01e3b65ce62235519eb09b26a0b3bb79e03d6057711e9e0d2e7651c3c9e339/cgroup.procs: no such file or directory: unknown"
E1206 06:31:12.070300      45 kuberuntime_sandbox.go:72] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: unable to apply cgroup configuration: failed to write 4507: write /sys/fs/cgroup/kubepods/besteffort/pod0a22ad1e-6958-416b-8f43-1533890c66eb/8a01e3b65ce62235519eb09b26a0b3bb79e03d6057711e9e0d2e7651c3c9e339/cgroup.procs: no such file or directory: unknown" pod="kube-system/local-path-provisioner-846b9dcb6c-wpv92"
E1206 06:31:12.070310      45 kuberuntime_manager.go:1168] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: unable to apply cgroup configuration: failed to write 4507: write /sys/fs/cgroup/kubepods/besteffort/pod0a22ad1e-6958-416b-8f43-1533890c66eb/8a01e3b65ce62235519eb09b26a0b3bb79e03d6057711e9e0d2e7651c3c9e339/cgroup.procs: no such file or directory: unknown" pod="kube-system/local-path-provisioner-846b9dcb6c-wpv92"
E1206 06:31:12.070337      45 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"local-path-provisioner-846b9dcb6c-wpv92_kube-system(0a22ad1e-6958-416b-8f43-1533890c66eb)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"local-path-provisioner-846b9dcb6c-wpv92_kube-system(0a22ad1e-6958-416b-8f43-1533890c66eb)\\\": rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: unable to apply cgroup configuration: failed to write 4507: write /sys/fs/cgroup/kubepods/besteffort/pod0a22ad1e-6958-416b-8f43-1533890c66eb/8a01e3b65ce62235519eb09b26a0b3bb79e03d6057711e9e0d2e7651c3c9e339/cgroup.procs: no such file or directory: unknown\"" pod="kube-system/local-path-provisioner-846b9dcb6c-wpv92" podUID="0a22ad1e-6958-416b-8f43-1533890c66eb"
W1206 06:31:15.161143      45 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods/besteffort/pod0a22ad1e-6958-416b-8f43-1533890c66eb/8a01e3b65ce62235519eb09b26a0b3bb79e03d6057711e9e0d2e7651c3c9e339 WatchSource:0}: container "8a01e3b65ce62235519eb09b26a0b3bb79e03d6057711e9e0d2e7651c3c9e339" in namespace "k8s.io": not found

Additional Information

No response

Inspect modules/k3s/src/main/java/org/testcontainers/k3s/K3sContainer.java around line 42, focusing on the deprecated method call identified in the report. Reproduce the K3sContainer startup on Podman ARM and compare the cgroup errors and pod states before and after the call; done means the system pods reach Running without the reported sandbox failures.