testcontainers / testcontainers/testcontainers-java

[Bug]: k3s container is unable to start on wsl2

Open
#10,375 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

type/bug
Dominant language
Java
Stars
8.7k
Forks
1.9k
Avg merge
2d 17h
Merged PRs (30d)
9

Description

Module

K3S

Testcontainers version

1.20.3

Using the latest Testcontainers version?

Yes

Host OS

Windows 11 WSL2

Host Arch

x64

Docker version
Client:
 Version:           28.1.1-rd
 API version:       1.47 (downgraded from 1.49)
 Go version:        go1.23.8
 Git commit:        4d7f01e
 Built:             Fri Apr 18 18:11:58 2025
 OS/Arch:           linux/amd64
 Context:           default

Server:
 Engine:
  Version:          27.3.1
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.23.9
  Git commit:       41ca978a0a5400cc24b274137efa9f25517fcc0b
  Built:            Thu May  8 20:02:17 2025
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v2.0.0
  GitCommit:        207ad711eabd375a01713109a8a197d197ff6542
 runc:
  Version:          1.2.2
  GitCommit:        7cb363254b69e10320360b63fb73e0ffb5da7bf2
 docker-init:
  Version:          0.19.0
  GitCommit:
What happened?

Context:

Using K3sContainer with Testcontainers to start an embedded cluster for integration tests. Some pods will be spawned for test purpose via the Operator SDK framework. We are using the org.testcontainers:junit-jupiter and org.testcontainers:k3s dependencies.

Rancher Desktop is providing a docker compatible API via its moby container engine. We have this setup on multiple machines using Windows 11 x64 and Mac.

Expected Result:

The pods should be able to start without any problems on different OS.

Actual Result:

The pods are unable to start due to issues with the cgroup system. At least on Windows. Tests on a Mac machine work without problems.

Existence of a workaround:

I have found a workaround in order to proceed with my daily tasks. Please find a more detailed explanation of it below.

Relevant log output
2025-06-27 10:27:21,578 INFO  [org.tes.DockerClientFactory] (main) Testcontainers version: 1.20.6
2025-06-27 10:27:21,657 INFO  [org.tes.doc.DockerClientProviderStrategy] (main) Loaded org.testcontainers.dockerclient.UnixSocketClientProviderStrategy from ~/.testcontainers.properties, will try it first
2025-06-27 10:27:21,803 INFO  [org.tes.doc.DockerClientProviderStrategy] (main) Found Docker environment with local Unix socket (unix:///var/run/docker.sock)
2025-06-27 10:27:21,824 INFO  [org.tes.DockerClientFactory] (pool-2-thread-1) Docker host IP address is localhost
2025-06-27 10:27:21,836 INFO  [org.tes.DockerClientFactory] (pool-2-thread-1) Connected to docker: 
  Server Version: 27.3.1
  API Version: 1.47
  Operating System: Rancher Desktop WSL Distribution
  Total Memory: 40109 MB
2025-06-27 10:27:21,875 INFO  [tc.tes.11.0] (pool-2-thread-1) Creating container for image: testcontainers/ryuk:0.11.0
2025-06-27 10:27:21,954 INFO  [org.tes.uti.RegistryAuthLocator] (pool-2-thread-1) Credential helper/store (docker-credential-wincred.exe) does not have credentials for https://index.docker.io/v1/
2025-06-27 10:27:22,072 INFO  [tc.tes.11.0] (pool-2-thread-1) Container testcontainers/ryuk:0.11.0 is starting: 3e33972713c713daee535d3c006e1f0ee6d343ac2afbe2acd63998bcbdb86377
2025-06-27 10:27:22,415 INFO  [tc.tes.11.0] (pool-2-thread-1) Container testcontainers/ryuk:0.11.0 started in PT0.540254449S
2025-06-27 10:27:22,417 INFO  [org.tes.uti.RyukResourceReaper] (pool-2-thread-1) Ryuk started - will monitor and terminate Testcontainers containers on JVM exit
2025-06-27 10:27:22,418 INFO  [org.tes.DockerClientFactory] (pool-2-thread-1) Checking the system...
2025-06-27 10:27:22,418 INFO  [org.tes.DockerClientFactory] (pool-2-thread-1) ✔︎ Docker server version should be at least 1.6.0
2025-06-27 10:27:22,418 INFO  [tc.ran.33.1-k3s1] (pool-2-thread-1) Creating container for image: rancher/k3s:v1.33.1-k3s1
2025-06-27 10:27:22,764 WARN  [org.tes.con.ContainerDef] (pool-2-thread-1) Unable to mount a file from test host into a running container. This may be a misconfiguration or limitation of your Docker environment. Some features might not work.
2025-06-27 10:27:22,808 INFO  [tc.ran.33.1-k3s1] (pool-2-thread-1) Container rancher/k3s:v1.33.1-k3s1 is starting: 4b25df6ebe50bc5f3a374e6cc3816e8c660f9e690a08ff12c01debd22659ab00
K3S: time="2025-06-27T08:27:23Z" level=info msg="Starting k3s v1.33.1+k3s1 (99d91538)"
K3S: time="2025-06-27T08:27:23Z" level=info msg="Configuring sqlite3 database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
K3S: time="2025-06-27T08:27:23Z" level=info msg="Configuring database table schema and indexes, this may take a moment..."
K3S: time="2025-06-27T08:27:23Z" level=info msg="Database tables and indexes are up to date"
K3S: time="2025-06-27T08:27:23Z" level=info msg="Kine available at unix://kine.sock"
.........
.........
K3S: I0627 08:27:36.853024      55 pod_container_manager_linux.go:207] "Failed to kill all the processes attached to cgroup" cgroupName=["kubepods","besteffort","pod7f6c53b6-16fc-44b3-a91c-f54a6e453779"] err="os: process not initialized"
K3S: I0627 08:27:36.853042      55 pod_container_manager_linux.go:207] "Failed to kill all the processes attached to cgroup" cgroupName=["kubepods","burstable","pod251b8724-2fba-43a7-94a1-e000139f6723"] err="os: process not initialized"
K3S: I0627 08:27:36.854615      55 pod_container_manager_linux.go:207] "Failed to kill all the processes attached to cgroup" cgroupName=["kubepods","burstable","poda2fabe93-9955-4f3e-83a6-8b3d7e95e816"] err="os: process not initialized"
K3S: time="2025-06-27T08:27:37Z" level=info msg="Starting network policy controller version v2.2.1, built on 2025-05-23T19:10:39Z, go1.24.2"
K3S: I0627 08:27:37.542464      55 network_policy_controller.go:164] Starting network policy controller
K3S: I0627 08:27:37.560735      55 network_policy_controller.go:176] Starting network policy controller full sync goroutine
K3S: time="2025-06-27T08:27:37Z" level=info msg="Started tunnel to 172.17.0.3:6443"
K3S: time="2025-06-27T08:27:37Z" level=info msg="Stopped tunnel to 127.0.0.1:6443"
K3S: time="2025-06-27T08:27:37Z" level=info msg="Proxy done" err="context canceled" url="wss://127.0.0.1:6443/v1-k3s/connect"
K3S: time="2025-06-27T08:27:37Z" level=info msg="Connecting to proxy" url="wss://172.17.0.3:6443/v1-k3s/connect"
K3S: time="2025-06-27T08:27:37Z" level=info msg="error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF"
K3S: time="2025-06-27T08:27:37Z" level=info msg="Handling backend connection request [4b25df6ebe50]"
K3S: time="2025-06-27T08:27:37Z" level=info msg="Remotedialer connected to proxy" url="wss://172.17.0.3:6443/v1-k3s/connect"

K3S: I0627 08:27:38.848375      55 pod_container_manager_linux.go:207] "Failed to kill all the processes attached to cgroup" cgroupName=["kubepods","burstable","pod251b8724-2fba-43a7-94a1-e000139f6723"] err="os: process not initialized"
K3S: I0627 08:27:38.848863      55 pod_container_manager_linux.go:207] "Failed to kill all the processes attached to cgroup" cgroupName=["kubepods","besteffort","pod7f6c53b6-16fc-44b3-a91c-f54a6e453779"] err="os: process not initialized"
K3S: I0627 08:27:38.849176      55 pod_container_manager_linux.go:207] "Failed to kill all the processes attached to cgroup" cgroupName=["kubepods","burstable","poda2fabe93-9955-4f3e-83a6-8b3d7e95e816"] err="os: process not initialized"
K3S: I0627 08:27:38.854361      55 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49b5e9da6608cf77e641373a299ac9a392c2ad9fb7fe25fbe4b7ecce5cf35656"
K3S: I0627 08:27:38.855340      55 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fb9ba02d6ff5fc379d9c4b9bcc302d056c485172bc2f3c699c09bce5a527da5"
K3S: I0627 08:27:38.856644      55 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23d6a099babf6955da1c400863b285d96c9e55fabdce0eb81a6f894c93d30d0b"
K3S: I0627 08:27:40.847543      55 pod_container_manager_linux.go:207] "Failed to kill all the processes attached to cgroup" cgroupName=["kubepods","burstable","pod251b8724-2fba-43a7-94a1-e000139f6723"] err="os: process not initialized"
K3S: I0627 08:27:40.847739      55 pod_container_manager_linux.go:207] "Failed to kill all the processes attached to cgroup" cgroupName=["kubepods","burstable","poda2fabe93-9955-4f3e-83a6-8b3d7e95e816"] err="os: process not initialized"
K3S: I0627 08:27:40.848011      55 pod_container_manager_linux.go:207] "Failed to kill all the processes attached to cgroup" cgroupName=["kubepods","besteffort","pod7f6c53b6-16fc-44b3-a91c-f54a6e453779"] err="os: process not initialized"
K3S: I0627 08:27:41.205541      55 kuberuntime_manager.go:1746] "Updating runtime config through cri with podcidr" CIDR="10.42.0.0/24"
K3S: I0627 08:27:41.206069      55 kubelet_network.go:61] "Updating Pod CIDR" originalPodCIDR="" newPodCIDR="10.42.0.0/24"
K3S: E0627 08:27:41.795322      55 log.go:32] "StartContainer from runtime service failed" err="rpc error: code = Unknown desc = sandbox container \"49b5e9da6608cf77e641373a299ac9a392c2ad9fb7fe25fbe4b7ecce5cf35656\" is not running" containerID="2146dc8d1fb34c77af0d10a1d999ec4e89ec438da92edbbf493bc13d894792bf"
K3S: E0627 08:27:41.795420      55 kuberuntime_manager.go:1358] "Unhandled Error" err="container &Container{Name:local-path-provisioner,Image:rancher/local-path-provisioner:v0.0.31,Command:[local-path-provisioner start --config /etc/config/config.json],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:POD_NAMESPACE,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},Claims:[]ResourceClaim{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:config-volume,ReadOnly:false,MountPath:/etc/config/,SubPath:,MountPropagation:nil,SubPathExpr:,RecursiveReadOnly:nil,},VolumeMount{Name:kube-api-access-z954w,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,RecursiveReadOnly:nil,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,ResizePolicy:[]ContainerResizePolicy{},RestartPolicy:nil,} start failed in pod local-path-provisioner-774c6665dc-hc695_kube-system(3a982885-c4ad-416d-9fff-ac92189b24fd): RunContainerError: sandbox container \"49b5e9da6608cf77e641373a299ac9a392c2ad9fb7fe25fbe4b7ecce5cf35656\" is not running"
K3S: E0627 08:27:41.796493      55 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"local-path-provisioner\" with RunContainerError: \"sandbox container \\\"49b5e9da6608cf77e641373a299ac9a392c2ad9fb7fe25fbe4b7ecce5cf35656\\\" is not running\"" pod="kube-system/local-path-provisioner-774c6665dc-hc695" podUID="3a982885-c4ad-416d-9fff-ac92189b24fd"
K3S: E0627 08:27:42.698177      55 log.go:32] "StartContainer from runtime service failed" err="rpc error: code = Unknown desc = sandbox container \"4fb9ba02d6ff5fc379d9c4b9bcc302d056c485172bc2f3c699c09bce5a527da5\" is not running" containerID="e5417a211a69f5ec95c7360f02c069e39b3feb36ebe51c5a6283353270c814ec"
K3S: E0627 08:27:42.698282      55 kuberuntime_manager.go:1358] "Unhandled Error" err="container &Container{Name:metrics-server,Image:rancher/mirrored-metrics-server:v0.7.2,Command:[],Args:[--cert-dir=/tmp --secure-port=10250 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --kubelet-use-node-status-port --metric-resolution=15s --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305],WorkingDir:,Ports:[]ContainerPort{ContainerPort{Name:https,HostPort:0,ContainerPort:10250,Protocol:TCP,HostIP:,},},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{cpu: {{100 -3} {<nil>} 100m DecimalSI},memory: {{73400320 0} {<nil>} 70Mi BinarySI},},Claims:[]ResourceClaim{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:tmp-dir,ReadOnly:false,MountPath:/tmp,SubPath:,MountPropagation:nil,SubPathExpr:,RecursiveReadOnly:nil,},VolumeMount{Name:kube-api-access-hr9bv,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,RecursiveReadOnly:nil,},},LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/livez,Port:{1 0 https},Host:,Scheme:HTTPS,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,TerminationGracePeriodSeconds:nil,},ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/readyz,Port:{1 0 https},Host:,Scheme:HTTPS,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:0,TimeoutSeconds:1,PeriodSeconds:2,SuccessThreshold:1,FailureThreshold:3,TerminationGracePeriodSeconds:nil,},Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:&SecurityContext{Capabilities:nil,Privileged:nil,SELinuxOptions:nil,RunAsUser:*1000,RunAsNonRoot:*true,ReadOnlyRootFilesystem:*true,AllowPrivilegeEscalation:*false,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:nil,AppArmorProfile:nil,},Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,ResizePolicy:[]ContainerResizePolicy{},RestartPolicy:nil,} start failed in pod metrics-server-6f4c6675d5-s7pp2_kube-system(ec1dcbc4-20fa-4372-b85c-be5348d10a84): RunContainerError: sandbox container \"4fb9ba02d6ff5fc379d9c4b9bcc302d056c485172bc2f3c699c09bce5a527da5\" is not running"
K3S: E0627 08:27:42.699384      55 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"metrics-server\" with RunContainerError: \"sandbox container \\\"4fb9ba02d6ff5fc379d9c4b9bcc302d056c485172bc2f3c699c09bce5a527da5\\\" is not running\"" pod="kube-system/metrics-server-6f4c6675d5-s7pp2" podUID="ec1dcbc4-20fa-4372-b85c-be5348d10a84"
K3S: I0627 08:27:42.846544      55 pod_container_manager_linux.go:207] "Failed to kill all the processes attached to cgroup" cgroupName=["kubepods","besteffort","pod7f6c53b6-16fc-44b3-a91c-f54a6e453779"] err="os: process not initialized"
K3S: I0627 08:27:42.846603      55 pod_container_manager_linux.go:207] "Failed to kill all the processes attached to cgroup" cgroupName=["kubepods","burstable","poda2fabe93-9955-4f3e-83a6-8b3d7e95e816"] err="os: process not initialized"
K3S: I0627 08:27:42.846808      55 pod_container_manager_linux.go:207] "Failed to kill all the processes attached to cgroup" cgroupName=["kubepods","burstable","pod251b8724-2fba-43a7-94a1-e000139f6723"] err="os: process not initialized"
K3S: I0627 08:27:42.868209      55 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f249b2642f91a93ed9eca12a6e4b0f9b57e0f699d2248988bc9eb2c02346d69"
K3S: I0627 08:27:42.868261      55 scope.go:117] "RemoveContainer" containerID="2146dc8d1fb34c77af0d10a1d999ec4e89ec438da92edbbf493bc13d894792bf"
K3S: E0627 08:27:42.876097      55 log.go:32] "StartContainer from runtime service failed" err="rpc error: code = Unknown desc = sandbox container \"23d6a099babf6955da1c400863b285d96c9e55fabdce0eb81a6f894c93d30d0b\" is not running" containerID="4722bb32891fd3736ba4173a397d05aaaf1b0ebead92a6ff0298e55dcc498180"
K3S: E0627 08:27:42.876339      55 kuberuntime_manager.go:1358] "Unhandled Error" err="container &Container{Name:coredns,Image:rancher/mirrored-coredns-coredns:1.12.1,Command:[],Args:[-conf /etc/coredns/Corefile],WorkingDir:,Ports:[]ContainerPort{ContainerPort{Name:dns,HostPort:0,ContainerPort:53,Protocol:UDP,HostIP:,},ContainerPort{Name:dns-tcp,HostPort:0,ContainerPort:53,Protocol:TCP,HostIP:,},ContainerPort{Name:metrics,HostPort:0,ContainerPort:9153,Protocol:TCP,HostIP:,},},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{memory: {{178257920 0} {<nil>} 170Mi BinarySI},},Requests:ResourceList{cpu: {{100 -3} {<nil>} 100m DecimalSI},memory: {{73400320 0} {<nil>} 70Mi BinarySI},},Claims:[]ResourceClaim{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:config-volume,ReadOnly:true,MountPath:/etc/coredns,SubPath:,MountPropagation:nil,SubPathExpr:,RecursiveReadOnly:nil,},VolumeMount{Name:custom-config-volume,ReadOnly:true,MountPath:/etc/coredns/custom,SubPath:,MountPropagation:nil,SubPathExpr:,RecursiveReadOnly:nil,},VolumeMount{Name:kube-api-access-wmtv6,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,RecursiveReadOnly:nil,},},LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/health,Port:{0 8080 },Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,TerminationGracePeriodSeconds:nil,},ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/ready,Port:{0 8181 },Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:0,TimeoutSeconds:1,PeriodSeconds:2,SuccessThreshold:1,FailureThreshold:3,TerminationGracePeriodSeconds:nil,},Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:&SecurityContext{Capabilities:&Capabilities{Add:[NET_BIND_SERVICE],Drop:[all],},Privileged:nil,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:*true,AllowPrivilegeEscalation:*false,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:nil,AppArmorProfile:nil,},Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,ResizePolicy:[]ContainerResizePolicy{},RestartPolicy:nil,} start failed in pod coredns-697968c856-92jdv_kube-system(6b220854-264f-4415-8c7d-d3ca42872cd3): RunContainerError: sandbox container \"23d6a099babf6955da1c400863b285d96c9e55fabdce0eb81a6f894c93d30d0b\" is not running"
K3S: E0627 08:27:42.877475      55 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"coredns\" with RunContainerError: \"sandbox container \\\"23d6a099babf6955da1c400863b285d96c9e55fabdce0eb81a6f894c93d30d0b\\\" is not running\"" pod="kube-system/coredns-697968c856-92jdv" podUID="6b220854-264f-4415-8c7d-d3ca42872cd3"
K3S: E0627 08:27:43.074768      55 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"local-path-provisioner\" with CrashLoopBackOff: \"back-off 10s restarting failed container=local-path-provisioner pod=local-path-provisioner-774c6665dc-hc695_kube-system(3a982885-c4ad-416d-9fff-ac92189b24fd)\"" pod="kube-system/local-path-provisioner-774c6665dc-hc695" podUID="3a982885-c4ad-416d-9fff-ac92189b24fd"
K3S: I0627 08:27:43.874658      55 scope.go:117] "RemoveContainer" containerID="7fe28a8188a9eaa436e49ff17f0675eba6cef527dbe1fcce04a0e3b6a1cb5e1b"
K3S: E0627 08:27:43.874732      55 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"local-path-provisioner\" with CrashLoopBackOff: \"back-off 10s restarting failed container=local-path-provisioner pod=local-path-provisioner-774c6665dc-hc695_kube-system(3a982885-c4ad-416d-9fff-ac92189b24fd)\"" pod="kube-system/local-path-provisioner-774c6665dc-hc695" podUID="3a982885-c4ad-416d-9fff-ac92189b24fd"
K3S: I0627 08:27:43.876266      55 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/metrics-server-6f4c6675d5-s7pp2" podStartSLOduration=3.064503207 podStartE2EDuration="7.876256355s" podCreationTimestamp="2025-06-27 08:27:36 +0000 UTC" firstStartedPulling="2025-06-27 08:27:37.871928407 +0000 UTC m=+13.919543984" lastFinishedPulling="2025-06-27 08:27:42.683681551 +0000 UTC m=+18.731297132" observedRunningTime="2025-06-27 08:27:43.876010156 +0000 UTC m=+19.923625740" watchObservedRunningTime="2025-06-27 08:27:43.876256355 +0000 UTC m=+19.923871940"
K3S: I0627 08:27:44.276498      55 controller.go:667] quota admission added evaluator for: jobs.batch
K3S: I0627 08:27:44.470449      55 alloc.go:328] "allocated clusterIPs" service="test/nginx-service" clusterIPs={"IPv4":"10.43.71.84"}
K3S: I0627 08:27:44.473612      55 cidrallocator.go:277] updated ClusterIP allocator for Service CIDR 10.43.0.0/16
K3S: I0627 08:27:44.559193      55 alloc.go:328] "allocated clusterIPs" service="test/keycloak-service" clusterIPs={"IPv4":"10.43.69.208"}
.........
.........
K3S: I0627 08:27:46.897461      55 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3f78c4978c38e0b082ec72eaef6cae4ddd80ef47cc0cc75cd7b45eb277f84be"
K3S: I0627 08:27:46.898023      55 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfa9a75f85361f5bca946233845b472665d8c21ce87ef99b73bb2ea9f3aa7f7f"
K3S: I0627 08:27:46.898832      55 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8838ddfec5b2f44071e18d5d26a6865ec4930871debc6d638652e2c5e6a73c3a"
K3S: I0627 08:27:46.899382      55 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e295afd6569f5f7105c2ea845a28e944f9e3a7b8d460c6571ff3c13872f6e67"
K3S: I0627 08:27:46.901122      55 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aec42fa38c1f131f7a195d5f52fb4d9faef6b7f3aac0d1c7dfa9658fd3010b76"
K3S: I0627 08:27:46.902663      55 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4410eb5dc71622d7920973c2e4ced1715162e5e1c47a3483dcaf1abe04a80a0f"
K3S: I0627 08:27:46.904963      55 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cac716960ca152d3f43512d38fd90f3d6553454ecc41b9c83d802d4cd0baaba0"
K3S: I0627 08:27:46.906109      55 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c33e5bb371fff9b5360d696d0bdc5350ab424ef7e1d8ae7d8e4964da6b64cc8"
K3S: I0627 08:27:46.907367      55 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a05e04c6fd0fc111ed0d77871cada5d05dd8bf187c6c65a7d786e14bb277b50"
K3S: E0627 08:27:47.159471      55 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"metrics-server\" with CrashLoopBackOff: \"back-off 10s restarting failed container=metrics-server pod=metrics-server-6f4c6675d5-s7pp2_kube-system(ec1dcbc4-20fa-4372-b85c-be5348d10a84)\"" pod="kube-system/metrics-server-6f4c6675d5-s7pp2" podUID="ec1dcbc4-20fa-4372-b85c-be5348d10a84"
K3S: E0627 08:27:47.160876      55 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"coredns\" with CrashLoopBackOff: \"back-off 10s restarting failed container=coredns pod=coredns-697968c856-92jdv_kube-system(6b220854-264f-4415-8c7d-d3ca42872cd3)\"" pod="kube-system/coredns-697968c856-92jdv" podUID="6b220854-264f-4415-8c7d-d3ca42872cd3"
K3S: E0627 08:27:47.188254      55 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"local-path-provisioner\" with CrashLoopBackOff: \"back-off 10s restarting failed container=local-path-provisioner pod=local-path-provisioner-774c6665dc-hc695_kube-system(3a982885-c4ad-416d-9fff-ac92189b24fd)\"" pod="kube-system/local-path-provisioner-774c6665dc-hc695" podUID="3a982885-c4ad-416d-9fff-ac92189b24fd"
K3S: I0627 08:27:47.911107      55 scope.go:117] "RemoveContainer" containerID="f9c3747fc3bc773aa9efa298d4e30f83c736abedc840b692224673486204e84a"
K3S: E0627 08:27:47.911213      55 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"coredns\" with CrashLoopBackOff: \"back-off 10s restarting failed container=coredns pod=coredns-697968c856-92jdv_kube-system(6b220854-264f-4415-8c7d-d3ca42872cd3)\"" pod="kube-system/coredns-697968c856-92jdv" podUID="6b220854-264f-4415-8c7d-d3ca42872cd3"
K3S: I0627 08:27:47.913610      55 scope.go:117] "RemoveContainer" containerID="7fe28a8188a9eaa436e49ff17f0675eba6cef527dbe1fcce04a0e3b6a1cb5e1b"
K3S: E0627 08:27:47.913732      55 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"local-path-provisioner\" with CrashLoopBackOff: \"back-off 10s restarting failed container=local-path-provisioner pod=local-path-provisioner-774c6665dc-hc695_kube-system(3a982885-c4ad-416d-9fff-ac92189b24fd)\"" pod="kube-system/local-path-provisioner-774c6665dc-hc695" podUID="3a982885-c4ad-416d-9fff-ac92189b24fd"
K3S: I0627 08:27:47.915773      55 scope.go:117] "RemoveContainer" containerID="408a940c4cbc78223a6030a619cc15d9cf018879aa1a8b57a40f44982dfffeca"
K3S: E0627 08:27:47.915871      55 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"metrics-server\" with CrashLoopBackOff: \"back-off 10s restarting failed container=metrics-server pod=metrics-server-6f4c6675d5-s7pp2_kube-system(ec1dcbc4-20fa-4372-b85c-be5348d10a84)\"" pod="kube-system/metrics-server-6f4c6675d5-s7pp2" podUID="ec1dcbc4-20fa-4372-b85c-be5348d10a84"
K3S: I0627 08:27:48.846902      55 pod_container_manager_linux.go:207] "Failed to kill all the processes attached to cgroup" cgroupName=["kubepods","besteffort","pod7f6c53b6-16fc-44b3-a91c-f54a6e453779"] err="os: process not initialized"
K3S: I0627 08:27:48.847097      55 pod_container_manager_linux.go:207] "Failed to kill all the processes attached to cgroup" cgroupName=["kubepods","burstable","poda2fabe93-9955-4f3e-83a6-8b3d7e95e816"] err="os: process not initialized"
K3S: I0627 08:27:48.847141      55 pod_container_manager_linux.go:207] "Failed to kill all the processes attached to cgroup" cgroupName=["kubepods","burstable","pod251b8724-2fba-43a7-94a1-e000139f6723"] err="os: process not initialized"
K3S: I0627 08:27:48.919326      55 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e6aa8893a3b7d4bf57b875ba5fd998ddad5cb7c10f3bdf510a829ee65bcc438"
K3S: I0627 08:27:48.921776      55 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3250ab0073eb2a632218e74301c0edc08a4522359ecd846cc591019fdc839eaf"
K3S: I0627 08:27:48.923992      55 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9d05acb0dd9a5332b8010ccea38d9c10a87816c546c34efdb7c53f30602c452"
K3S: E0627 08:27:49.163430      55 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"metrics-server\" with CrashLoopBackOff: \"back-off 10s restarting failed container=metrics-server pod=metrics-server-6f4c6675d5-s7pp2_kube-system(ec1dcbc4-20fa-4372-b85c-be5348d10a84)\"" pod="kube-system/metrics-server-6f4c6675d5-s7pp2" podUID="ec1dcbc4-20fa-4372-b85c-be5348d10a84"
K3S: E0627 08:27:49.166339      55 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"coredns\" with CrashLoopBackOff: \"back-off 10s restarting failed container=coredns pod=coredns-697968c856-92jdv_kube-system(6b220854-264f-4415-8c7d-d3ca42872cd3)\"" pod="kube-system/coredns-697968c856-92jdv" podUID="6b220854-264f-4415-8c7d-d3ca42872cd3"
K3S: E0627 08:27:49.167638      55 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"local-path-provisioner\" with CrashLoopBackOff: \"back-off 10s restarting failed container=local-path-provisioner pod=local-path-provisioner-774c6665dc-hc695_kube-system(3a982885-c4ad-416d-9fff-ac92189b24fd)\"" pod="kube-system/local-path-provisioner-774c6665dc-hc695" podUID="3a982885-c4ad-416d-9fff-ac92189b24fd"
K3S: I0627 08:27:49.928120      55 scope.go:117] "RemoveContainer" containerID="408a940c4cbc78223a6030a619cc15d9cf018879aa1a8b57a40f44982dfffeca"
K3S: E0627 08:27:49.928221      55 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"metrics-server\" with CrashLoopBackOff: \"back-off 10s restarting failed container=metrics-server pod=metrics-server-6f4c6675d5-s7pp2_kube-system(ec1dcbc4-20fa-4372-b85c-be5348d10a84)\"" pod="kube-system/metrics-server-6f4c6675d5-s7pp2" podUID="ec1dcbc4-20fa-4372-b85c-be5348d10a84"
K3S: I0627 08:27:49.930222      55 scope.go:117] "RemoveContainer" containerID="f9c3747fc3bc773aa9efa298d4e30f83c736abedc840b692224673486204e84a"
K3S: E0627 08:27:49.930302      55 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"coredns\" with CrashLoopBackOff: \"back-off 10s restarting failed container=coredns pod=coredns-697968c856-92jdv_kube-system(6b220854-264f-4415-8c7d-d3ca42872cd3)\"" pod="kube-system/coredns-697968c856-92jdv" podUID="6b220854-264f-4415-8c7d-d3ca42872cd3"
K3S: I0627 08:27:49.932312      55 scope.go:117] "RemoveContainer" containerID="7fe28a8188a9eaa436e49ff17f0675eba6cef527dbe1fcce04a0e3b6a1cb5e1b"
K3S: E0627 08:27:49.932403      55 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"local-path-provisioner\" with CrashLoopBackOff: \"back-off 10s restarting failed container=local-path-provisioner pod=local-path-provisioner-774c6665dc-hc695_kube-system(3a982885-c4ad-416d-9fff-ac92189b24fd)\"" pod="kube-system/local-path-provisioner-774c6665dc-hc695" podUID="3a982885-c4ad-416d-9fff-ac92189b24fd"
K3S: I0627 08:27:50.846372      55 pod_container_manager_linux.go:207] "Failed to kill all the processes attached to cgroup" cgroupName=["kubepods","burstable","poda2fabe93-9955-4f3e-83a6-8b3d7e95e816"] err="os: process not initialized"
K3S: I0627 08:27:50.846556      55 pod_container_manager_linux.go:207] "Failed to kill all the processes attached to cgroup" cgroupName=["kubepods","besteffort","pod7f6c53b6-16fc-44b3-a91c-f54a6e453779"] err="os: process not initialized"
K3S: I0627 08:27:50.846591      55 pod_container_manager_linux.go:207] "Failed to kill all the processes attached to cgroup" cgroupName=["kubepods","burstable","pod251b8724-2fba-43a7-94a1-e000139f6723"] err="os: process not initialized"
K3S: I0627 08:27:50.936492      55 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16838158f20e10e5fb36d4d30323cc37488870f28176666ff59bf8b850b94c1f"
K3S: I0627 08:27:50.938735      55 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e53946fa05ef4669bdbe9dd0029922d2dc88151d36fbfacbfa07bb2efa2f34ed"
K3S: I0627 08:27:50.941433      55 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89e5af92c8eee73b3606c9e9841ca331320291d96d3b1118316b2931336f57c4"
K3S: E0627 08:27:51.211902      55 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"coredns\" with CrashLoopBackOff: \"back-off 10s restarting failed container=coredns pod=coredns-697968c856-92jdv_kube-system(6b220854-264f-4415-8c7d-d3ca42872cd3)\"" pod="kube-system/coredns-697968c856-92jdv" podUID="6b220854-264f-4415-8c7d-d3ca42872cd3"
K3S: E0627 08:27:51.213127      55 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"local-path-provisioner\" with CrashLoopBackOff: \"back-off 10s restarting failed container=local-path-provisioner pod=local-path-provisioner-774c6665dc-hc695_kube-system(3a982885-c4ad-416d-9fff-ac92189b24fd)\"" pod="kube-system/local-path-provisioner-774c6665dc-hc695" podUID="3a982885-c4ad-416d-9fff-ac92189b24fd"
K3S: E0627 08:27:51.215220      55 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"metrics-server\" with CrashLoopBackOff: \"back-off 10s restarting failed container=metrics-server pod=metrics-server-6f4c6675d5-s7pp2_kube-system(ec1dcbc4-20fa-4372-b85c-be5348d10a84)\"" pod="kube-system/metrics-server-6f4c6675d5-s7pp2" podUID="ec1dcbc4-20fa-4372-b85c-be5348d10a84"
K3S: I0627 08:27:51.945099      55 scope.go:117] "RemoveContainer" containerID="408a940c4cbc78223a6030a619cc15d9cf018879aa1a8b57a40f44982dfffeca"
K3S: E0627 08:27:51.945207      55 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"metrics-server\" with CrashLoopBackOff: \"back-off 10s restarting failed container=metrics-server pod=metrics-server-6f4c6675d5-s7pp2_kube-system(ec1dcbc4-20fa-4372-b85c-be5348d10a84)\"" pod="kube-system/metrics-server-6f4c6675d5-s7pp2" podUID="ec1dcbc4-20fa-4372-b85c-be5348d10a84"
K3S: I0627 08:27:51.948040      55 scope.go:117] "RemoveContainer" containerID="f9c3747fc3bc773aa9efa298d4e30f83c736abedc840b692224673486204e84a"
K3S: E0627 08:27:51.948144      55 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"coredns\" with CrashLoopBackOff: \"back-off 10s restarting failed container=coredns pod=coredns-697968c856-92jdv_kube-system(6b220854-264f-4415-8c7d-d3ca42872cd3)\"" pod="kube-system/coredns-697968c856-92jdv" podUID="6b220854-264f-4415-8c7d-d3ca42872cd3"
K3S: I0627 08:27:51.950823      55 scope.go:117] "RemoveContainer" containerID="7fe28a8188a9eaa436e49ff17f0675eba6cef527dbe1fcce04a0e3b6a1cb5e1b"
K3S: E0627 08:27:51.950918      55 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"local-path-provisioner\" with CrashLoopBackOff: \"back-off 10s restarting failed container=local-path-provisioner pod=local-path-provisioner-774c6665dc-hc695_kube-system(3a982885-c4ad-416d-9fff-ac92189b24fd)\"" pod="kube-system/local-path-provisioner-774c6665dc-hc695" podUID="3a982885-c4ad-416d-9fff-ac92189b24fd"
K3S: I0627 08:27:52.846356      55 pod_container_manager_linux.go:207] "Failed to kill all the processes attached to cgroup" cgroupName=["kubepods","burstable","poda2fabe93-9955-4f3e-83a6-8b3d7e95e816"] err="os: process not initialized"
K3S: I0627 08:27:52.846512      55 pod_container_manager_linux.go:207] "Failed to kill all the processes attached to cgroup" cgroupName=["kubepods","besteffort","pod7f6c53b6-16fc-44b3-a91c-f54a6e453779"] err="os: process not initialized"
K3S: I0627 08:27:52.846642      55 pod_container_manager_linux.go:207] "Failed to kill all the processes attached to cgroup" cgroupName=["kubepods","burstable","pod251b8724-2fba-43a7-94a1-e000139f6723"] err="os: process not initialized"
K3S: I0627 08:27:52.954559      55 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0a41d4267eb10c130693291870c12c96c9dab9fb7b5f6219342545521795877"
K3S: I0627 08:27:52.957055      55 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66c7160928b46f3cb4f1d0e76c2b8174253fa5035504749bfb07a34f04fa301e"
K3S: I0627 08:27:52.959496      55 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e71a8899fd3268ae57f3221ea2a2ef6eb47782b79113f9f6ec7c937ada084fd"
K3S: E0627 08:27:53.244311      55 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"metrics-server\" with CrashLoopBackOff: \"back-off 10s restarting failed container=metrics-server pod=metrics-server-6f4c6675d5-s7pp2_kube-system(ec1dcbc4-20fa-4372-b85c-be5348d10a84)\"" pod="kube-system/metrics-server-6f4c6675d5-s7pp2" podUID="ec1dcbc4-20fa-4372-b85c-be5348d10a84"
K3S: E0627 08:27:53.248277      55 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"coredns\" with CrashLoopBackOff: \"back-off 10s restarting failed container=coredns pod=coredns-697968c856-92jdv_kube-system(6b220854-264f-4415-8c7d-d3ca42872cd3)\"" pod="kube-system/coredns-697968c856-92jdv" podUID="6b220854-264f-4415-8c7d-d3ca42872cd3"
K3S: I0627 08:27:53.963255      55 scope.go:117] "RemoveContainer" containerID="f9c3747fc3bc773aa9efa298d4e30f83c736abedc840b692224673486204e84a"
K3S: E0627 08:27:53.963357      55 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"coredns\" with CrashLoopBackOff: \"back-off 10s restarting failed container=coredns pod=coredns-697968c856-92jdv_kube-system(6b220854-264f-4415-8c7d-d3ca42872cd3)\"" pod="kube-system/coredns-697968c856-92jdv" podUID="6b220854-264f-4415-8c7d-d3ca42872cd3"
K3S: I0627 08:27:53.968827      55 scope.go:117] "RemoveContainer" containerID="408a940c4cbc78223a6030a619cc15d9cf018879aa1a8b57a40f44982dfffeca"
K3S: E0627 08:27:53.968956      55 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"metrics-server\" with CrashLoopBackOff: \"back-off 10s restarting failed container=metrics-server pod=metrics-server-6f4c6675d5-s7pp2_kube-system(ec1dcbc4-20fa-4372-b85c-be5348d10a84)\"" pod="kube-system/metrics-server-6f4c6675d5-s7pp2" podUID="ec1dcbc4-20fa-4372-b85c-be5348d10a84"
K3S: I0627 08:27:53.975871      55 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/local-path-provisioner-774c6665dc-hc695" podStartSLOduration=14.066817742 podStartE2EDuration="17.975857672s" podCreationTimestamp="2025-06-27 08:27:36 +0000 UTC" firstStartedPulling="2025-06-27 08:27:37.870164162 +0000 UTC m=+13.917779744" lastFinishedPulling="2025-06-27 08:27:41.779204093 +0000 UTC m=+17.826819674" observedRunningTime="2025-06-27 08:27:53.97508407 +0000 UTC m=+30.022699655" watchObservedRunningTime="2025-06-27 08:27:53.975857672 +0000 UTC m=+30.023473256"
K3S: I0627 08:27:54.847225      55 pod_container_manager_linux.go:207] "Failed to kill all the processes attached to cgroup" cgroupName=["kubepods","besteffort","pod7f6c53b6-16fc-44b3-a91c-f54a6e453779"] err="os: process not initialized"
K3S: I0627 08:27:54.847371      55 pod_container_manager_linux.go:207] "Failed to kill all the processes attached to cgroup" cgroupName=["kubepods","burstable","pod251b8724-2fba-43a7-94a1-e000139f6723"] err="os: process not initialized"
K3S: I0627 08:27:54.847371      55 pod_container_manager_linux.go:207] "Failed to kill all the processes attached to cgroup" cgroupName=["kubepods","burstable","poda2fabe93-9955-4f3e-83a6-8b3d7e95e816"] err="os: process not initialized"
K3S: I0627 08:27:54.972415      55 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d90d519ce6aa8aa51e5f5fc4c873c7bc8c0a3a849b585e3bdd4ef5f0a4ab3ffd"
K3S: I0627 08:27:54.975019      55 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54929ded732d3afdc1f297d9875d62075fecd4facfe2f269e3cdf705cfd517ba"
K3S: I0627 08:27:54.979207      55 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2ad6287498445f677ac031abfba583cdeae173f2accf941ee2b94fd8ac837b4"
K3S: I0627 08:27:54.979235      55 scope.go:117] "RemoveContainer" containerID="7fe28a8188a9eaa436e49ff17f0675eba6cef527dbe1fcce04a0e3b6a1cb5e1b"
K3S: E0627 08:27:55.259640      55 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"local-path-provisioner\" with CrashLoopBackOff: \"back-off 20s restarting failed container=local-path-provisioner pod=local-path-provisioner-774c6665dc-hc695_kube-system(3a982885-c4ad-416d-9fff-ac92189b24fd)\"" pod="kube-system/local-path-provisioner-774c6665dc-hc695" podUID="3a982885-c4ad-416d-9fff-ac92189b24fd"
K3S: E0627 08:27:55.338875      55 log.go:32] "StartContainer from runtime service failed" err="rpc error: code = Unknown desc = sandbox container \"f3f78c4978c38e0b082ec72eaef6cae4ddd80ef47cc0cc75cd7b45eb277f84be\" is not running" containerID="feda796405a649165ebe40ca7137af39ed4a3d9ab50c484fe284647aa1aa65bc"
K3S: E0627 08:27:55.338972      55 kuberuntime_manager.go:1358] "Unhandled Error" err="init container &Container{Name:fix-permissions,Image:public.ecr.aws/docker/library/busybox:latest,Command:[sh -c mkdir -p /opt/storage/shared/deployments && chown -R 1000:1000 /opt/storage/shared],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},Claims:[]ResourceClaim{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:storage,ReadOnly:false,MountPath:/opt/storage,SubPath:,MountPropagation:nil,SubPathExpr:,RecursiveReadOnly:nil,},VolumeMount{Name:kube-api-access-8rlns,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,RecursiveReadOnly:nil,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:Always,SecurityContext:&SecurityContext{Capabilities:nil,Privileged:nil,SELinuxOptions:nil,RunAsUser:*0,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:nil,AppArmorProfile:nil,},Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,ResizePolicy:[]ContainerResizePolicy{},RestartPolicy:nil,} start failed in pod storage-5486697b4d-t86qd_test(c8dbfece-6c57-43e2-b495-03e779a19f20): RunContainerError: sandbox container \"f3f78c4978c38e0b082ec72eaef6cae4ddd80ef47cc0cc75cd7b45eb277f84be\" is not running"
K3S: E0627 08:27:55.340051      55 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"fix-permissions\" with RunContainerError: \"sandbox container \\\"f3f78c4978c38e0b082ec72eaef6cae4ddd80ef47cc0cc75cd7b45eb277f84be\\\" is not running\"" pod="test/storage-5486697b4d-t86qd" podUID="c8dbfece-6c57-43e2-b495-03e779a19f20"
K3S: E0627 08:27:55.344607      55 log.go:32] "StartContainer from runtime service failed" err="rpc error: code = Unknown desc = sandbox container \"1c33e5bb371fff9b5360d696d0bdc5350ab424ef7e1d8ae7d8e4964da6b64cc8\" is not running" containerID="5cadd7ce36991593a9661781a2903baa0f031ca248a358201c61f1cbd5c2c786"
K3S: E0627 08:27:55.344703      55 kuberuntime_manager.go:1358] "Unhandled Error" err="container &Container{Name:storage-cleanup,Image:public.ecr.aws/docker/library/busybox,Command:[sh -c rm -rf /opt/storage/nodes/*],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},Claims:[]ResourceClaim{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:storage,ReadOnly:false,MountPath:/opt/storage,SubPath:,MountPropagation:nil,SubPathExpr:,RecursiveReadOnly:nil,},VolumeMount{Name:kube-api-access-pnvn6,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,RecursiveReadOnly:nil,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:Always,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,ResizePolicy:[]ContainerResizePolicy{},RestartPolicy:nil,} start failed in pod storage-cleanup-job-9b768_test(7f6646d8-8fe9-4659-aaf1-b26822e0f9d6): RunContainerError: sandbox container \"1c33e5bb371fff9b5360d696d0bdc5350ab424ef7e1d8ae7d8e4964da6b64cc8\" is not running"
K3S: E0627 08:27:55.345769      55 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-cleanup\" with RunContainerError: \"sandbox container \\\"1c33e5bb371fff9b5360d696d0bdc5350ab424ef7e1d8ae7d8e4964da6b64cc8\\\" is not running\"" pod="test/storage-cleanup-job-9b768" podUID="7f6646d8-8fe9-4659-aaf1-b26822e0f9d6"
K3S: I0627 08:27:55.984006      55 scope.go:117] "RemoveContainer" containerID="9bcaf19106ef3c5e8c3e74f5a1ec363eeffe9d1c11991b43568d0bed4941337e"
K3S: E0627 08:27:55.984125      55 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"local-path-provisioner\" with CrashLoopBackOff: \"back-off 20s restarting failed container=local-path-provisioner pod=local-path-provisioner-774c6665dc-hc695_kube-system(3a982885-c4ad-416d-9fff-ac92189b24fd)\"" pod="kube-system/local-path-provisioner-774c6665dc-hc695" podUID="3a982885-c4ad-416d-9fff-ac92189b24fd"
K3S: I0627 08:27:56.005994      55 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/coredns-697968c856-92jdv" podStartSLOduration=15.019436048 podStartE2EDuration="20.005982367s" podCreationTimestamp="2025-06-27 08:27:36 +0000 UTC" firstStartedPulling="2025-06-27 08:27:37.867311933 +0000 UTC m=+13.914927509" lastFinishedPulling="2025-06-27 08:27:42.853858246 +0000 UTC m=+18.901473828" observedRunningTime="2025-06-27 08:27:56.005809487 +0000 UTC m=+32.053425072" watchObservedRunningTime="2025-06-27 08:27:56.005982367 +0000 UTC m=+32.053597952"
K3S: I0627 08:27:56.064437      55 scope.go:117] "RemoveContainer" containerID="feda796405a649165ebe40ca7137af39ed4a3d9ab50c484fe284647aa1aa65bc"
K3S: I0627 08:27:56.847653      55 pod_container_manager_linux.go:207] "Failed to kill all the processes attached to cgroup" cgroupName=["kubepods","burstable","poda2fabe93-9955-4f3e-83a6-8b3d7e95e816"] err="os: process not initialized"
K3S: I0627 08:27:56.847653      55 pod_container_manager_linux.go:207] "Failed to kill all the processes attached to cgroup" cgroupName=["kubepods","burstable","pod251b8724-2fba-43a7-94a1-e000139f6723"] err="os: process not initialized"
K3S: I0627 08:27:56.847659      55 pod_container_manager_linux.go:207] "Failed to kill all the processes attached to cgroup" cgroupName=["kubepods","besteffort","pod7f6c53b6-16fc-44b3-a91c-f54a6e453779"] err="os: process not initialized"
K3S: I0627 08:27:56.998205      55 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="703febb2c0bcd8efdbf63305e67070cf88767336238d0ecb29b4e8b7eb86606e"
K3S: I0627 08:27:57.001274      55 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1041b9dfb009ca11372b5faaed8c9be0866b4bc21c6c3cb94969b7b63e02c357"
K3S: I0627 08:27:57.001296      55 scope.go:117] "RemoveContainer" containerID="408a940c4cbc78223a6030a619cc15d9cf018879aa1a8b57a40f44982dfffeca"
K3S: I0627 08:27:57.004653      55 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e198829ed327133bf3f3852b7350cfbe8014cb860c5c353f16767681d4a3d23d"
K3S: I0627 08:27:57.007469      55 scope.go:117] "RemoveContainer" containerID="f9c3747fc3bc773aa9efa298d4e30f83c736abedc840b692224673486204e84a"
K3S: I0627 08:27:57.009063      55 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba7aec7259f7ea440636c0cb5dbdf79dd49a1357e1204e8adae1928f82435dec"
K3S: I0627 08:27:57.184415      55 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pnvn6\" (UniqueName: \"kubernetes.io/projected/7f6646d8-8fe9-4659-aaf1-b26822e0f9d6-kube-api-access-pnvn6\") pod \"7f6646d8-8fe9-4659-aaf1-b26822e0f9d6\" (UID: \"7f6646d8-8fe9-4659-aaf1-b26822e0f9d6\") "
K3S: I0627 08:27:57.184441      55 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"storage\" (UniqueName: \"kubernetes.io/host-path/7f6646d8-8fe9-4659-aaf1-b26822e0f9d6-storage-volume-test\") pod \"7f6646d8-8fe9-4659-aaf1-b26822e0f9d6\" (UID: \"7f6646d8-8fe9-4659-aaf1-b26822e0f9d6\") "
K3S: I0627 08:27:57.184504      55 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/7f6646d8-8fe9-4659-aaf1-b26822e0f9d6-storage-volume-test" (OuterVolumeSpecName: "storage") pod "7f6646d8-8fe9-4659-aaf1-b26822e0f9d6" (UID: "7f6646d8-8fe9-4659-aaf1-b26822e0f9d6"). InnerVolumeSpecName "storage-volume-test". PluginName "kubernetes.io/host-path", VolumeGIDValue ""
K3S: I0627 08:27:57.186147      55 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f6646d8-8fe9-4659-aaf1-b26822e0f9d6-kube-api-access-pnvn6" (OuterVolumeSpecName: "kube-api-access-pnvn6") pod "7f6646d8-8fe9-4659-aaf1-b26822e0f9d6" (UID: "7f6646d8-8fe9-4659-aaf1-b26822e0f9d6"). InnerVolumeSpecName "kube-api-access-pnvn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
K3S: I0627 08:27:57.285169      55 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnvn6\" (UniqueName: \"kubernetes.io/projected/7f6646d8-8fe9-4659-aaf1-b26822e0f9d6-kube-api-access-pnvn6\") on node \"4b25df6ebe50\" DevicePath \"\""
K3S: I0627 08:27:57.285195      55 reconciler_common.go:299] "Volume detached for volume \"storage-volume-test\" (UniqueName: \"kubernetes.io/host-path/7f6646d8-8fe9-4659-aaf1-b26822e0f9d6-storage-volume-test\") on node \"4b25df6ebe50\" DevicePath \"\""
K3S: E0627 08:27:57.291016      55 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"local-path-provisioner\" with CrashLoopBackOff: \"back-off 20s restarting failed container=local-path-provisioner pod=local-path-provisioner-774c6665dc-hc695_kube-system(3a982885-c4ad-416d-9fff-ac92189b24fd)\"" pod="kube-system/local-path-provisioner-774c6665dc-hc695" podUID="3a982885-c4ad-416d-9fff-ac92189b24fd"
K3S: E0627 08:27:57.292375      55 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"metrics-server\" with CrashLoopBackOff: \"back-off 20s restarting failed container=metrics-server pod=metrics-server-6f4c6675d5-s7pp2_kube-system(ec1dcbc4-20fa-4372-b85c-be5348d10a84)\"" pod="kube-system/metrics-server-6f4c6675d5-s7pp2" podUID="ec1dcbc4-20fa-4372-b85c-be5348d10a84"
K3S: E0627 08:27:57.305376      55 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"coredns\" with CrashLoopBackOff: \"back-off 20s restarting failed container=coredns pod=coredns-697968c856-92jdv_kube-system(6b220854-264f-4415-8c7d-d3ca42872cd3)\"" pod="kube-system/coredns-697968c856-92jdv" podUID="6b220854-264f-4415-8c7d-d3ca42872cd3"
K3S: E0627 08:27:57.456082      55 log.go:32] "StartContainer from runtime service failed" err="rpc error: code = Unknown desc = sandbox container \"703febb2c0bcd8efdbf63305e67070cf88767336238d0ecb29b4e8b7eb86606e\" is not running" containerID="9666916fd7a16e7140fe85a4d21455bf164d53dc2d335055e579d3fd6977b857"
K3S: E0627 08:27:57.456166      55 kuberuntime_manager.go:1358] "Unhandled Error" err="init container &Container{Name:fix-permissions,Image:public.ecr.aws/docker/library/busybox:latest,Command:[sh -c mkdir -p /opt/storage/shared/deployments && chown -R 1000:1000 /opt/storage/shared],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},Claims:[]ResourceClaim{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:storage,ReadOnly:false,MountPath:/opt/storage,SubPath:,MountPropagation:nil,SubPathExpr:,RecursiveReadOnly:nil,},VolumeMount{Name:kube-api-access-8rlns,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,RecursiveReadOnly:nil,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:Always,SecurityContext:&SecurityContext{Capabilities:nil,Privileged:nil,SELinuxOptions:nil,RunAsUser:*0,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:nil,AppArmorProfile:nil,},Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,ResizePolicy:[]ContainerResizePolicy{},RestartPolicy:nil,} start failed in pod storage-5486697b4d-t86qd_test(c8dbfece-6c57-43e2-b495-03e779a19f20): RunContainerError: sandbox container \"703febb2c0bcd8efdbf63305e67070cf88767336238d0ecb29b4e8b7eb86606e\" is not running"
K3S: E0627 08:27:57.457260      55 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"fix-permissions\" with RunContainerError: \"sandbox container \\\"703febb2c0bcd8efdbf63305e67070cf88767336238d0ecb29b4e8b7eb86606e\\\" is not running\"" pod="test/storage-5486697b4d-t86qd" podUID="c8dbfece-6c57-43e2-b495-03e779a19f20"
K3S: E0627 08:27:57.940887      55 log.go:32] "StartContainer from runtime service failed" err="rpc error: code = Unknown desc = sandbox container \"0a05e04c6fd0fc111ed0d77871cada5d05dd8bf187c6c65a7d786e14bb277b50\" is not running" containerID="d4884e7710e17e472054bebaa76134573c6d236cfa3c4cfddd7764360af8ef75"
K3S: E0627 08:27:57.940982      55 kuberuntime_manager.go:1358] "Unhandled Error" err="init container &Container{Name:init-permissions,Image:registry.access.redhat.com/ubi8,Command:[sh -c chmod g+rws /opt/keycloak/data],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},Claims:[]ResourceClaim{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:storage,ReadOnly:false,MountPath:/opt/keycloak/data,SubPath:keycloak/data,MountPropagation:nil,SubPathExpr:,RecursiveReadOnly:nil,},VolumeMount{Name:kube-api-access-bcmmv,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,RecursiveReadOnly:nil,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:Always,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,ResizePolicy:[]ContainerResizePolicy{},RestartPolicy:nil,} start failed in pod keycloak-deployment-769c677ffc-ps2f4_test(9b972221-934f-49b0-8e50-a4707f900998): RunContainerError: sandbox container \"0a05e04c6fd0fc111ed0d77871cada5d05dd8bf187c6c65a7d786e14bb277b50\" is not running"
K3S: E0627 08:27:57.942079      55 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"init-permissions\" with RunContainerError: \"sandbox container \\\"0a05e04c6fd0fc111ed0d77871cada5d05dd8bf187c6c65a7d786e14bb277b50\\\" is not running\"" pod="test/keycloak-deployment-769c677ffc-ps2f4" podUID="9b972221-934f-49b0-8e50-a4707f900998"
K3S: I0627 08:27:58.013297      55 scope.go:117] "RemoveContainer" containerID="83d57cf3e03839e44f062c137005ceff3f84181ef6afe5ffa7a8fe5186437970"
K3S: E0627 08:27:58.013409      55 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"metrics-server\" with CrashLoopBackOff: \"back-off 20s restarting failed container=metrics-server pod=metrics-server-6f4c6675d5-s7pp2_kube-system(ec1dcbc4-20fa-4372-b85c-be5348d10a84)\"" pod="kube-system/metrics-server-6f4c6675d5-s7pp2" podUID="ec1dcbc4-20fa-4372-b85c-be5348d10a84"
K3S: I0627 08:27:58.016372      55 scope.go:117] "RemoveContainer" containerID="483d75b4c5e43ab774eb1b5497e4ae5d11309c95994aaedcc79a555cdff72fe7"
K3S: E0627 08:27:58.016576      55 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"coredns\" with CrashLoopBackOff: \"back-off 20s restarting failed container=coredns pod=coredns-697968c856-92jdv_kube-system(6b220854-264f-4415-8c7d-d3ca42872cd3)\"" pod="kube-system/coredns-697968c856-92jdv" podUID="6b220854-264f-4415-8c7d-d3ca42872cd3"
K3S: I0627 08:27:58.020508      55 scope.go:117] "RemoveContainer" containerID="9bcaf19106ef3c5e8c3e74f5a1ec363eeffe9d1c11991b43568d0bed4941337e"
K3S: E0627 08:27:58.020609      55 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"local-path-provisioner\" with CrashLoopBackOff: \"back-off 20s restarting failed container=local-path-provisioner pod=local-path-provisioner-774c6665dc-hc695_kube-system(3a982885-c4ad-416d-9fff-ac92189b24fd)\"" pod="kube-system/local-path-provisioner-774c6665dc-hc695" podUID="3a982885-c4ad-416d-9fff-ac92189b24fd"
K3S: I0627 08:27:58.164305      55 scope.go:117] "RemoveContainer" containerID="9666916fd7a16e7140fe85a4d21455bf164d53dc2d335055e579d3fd6977b857"
K3S: I0627 08:27:58.201032      55 scope.go:117] "RemoveContainer" containerID="d4884e7710e17e472054bebaa76134573c6d236cfa3c4cfddd7764360af8ef75"
K3S: E0627 08:27:58.283484      55 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"fix-permissions\" with CrashLoopBackOff: \"back-off 10s restarting failed container=fix-permissions pod=storage-5486697b4d-t86qd_test(c8dbfece-6c57-43e2-b495-03e779a19f20)\"" pod="test/storage-5486697b4d-t86qd" podUID="c8dbfece-6c57-43e2-b495-03e779a19f20"
K3S: I0627 08:27:58.846482      55 pod_container_manager_linux.go:207] "Failed to kill all the processes attached to cgroup" cgroupName=["kubepods","burstable","pod251b8724-2fba-43a7-94a1-e000139f6723"] err="os: process not initialized"
K3S: I0627 08:27:58.846518      55 pod_container_manager_linux.go:207] "Failed to kill all the processes attached to cgroup" cgroupName=["kubepods","besteffort","pod7f6c53b6-16fc-44b3-a91c-f54a6e453779"] err="os: process not initialized"
K3S: I0627 08:27:58.846573      55 pod_container_manager_linux.go:207] "Failed to kill all the processes attached to cgroup" cgroupName=["kubepods","burstable","poda2fabe93-9955-4f3e-83a6-8b3d7e95e816"] err="os: process not initialized"
K3S: E0627 08:27:58.925001      55 log.go:32] "StartContainer from runtime service failed" err="rpc error: code = Unknown desc = sandbox container \"1dc34867fe06c2dadbbe2f4a2f923f4ff947abadc52ed567e2a5f2240670a4ea\" is not running" containerID="eafd27bae0981a09d8f1be86f1a3224cfd79fffe3ef482ddd43784c0c12fce65"
K3S: E0627 08:27:58.925085      55 kuberuntime_manager.go:1358] "Unhandled Error" err="init container &Container{Name:init-permissions,Image:registry.access.redhat.com/ubi8,Command:[sh -c chmod g+rws /opt/keycloak/data],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},Claims:[]ResourceClaim{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:storage,ReadOnly:false,MountPath:/opt/keycloak/data,SubPath:keycloak/data,MountPropagation:nil,SubPathExpr:,RecursiveReadOnly:nil,},VolumeMount{Name:kube-api-access-bcmmv,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,RecursiveReadOnly:nil,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:Always,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,ResizePolicy:[]ContainerResizePolicy{},RestartPolicy:nil,} start failed in pod keycloak-deployment-769c677ffc-ps2f4_test(9b972221-934f-49b0-8e50-a4707f900998): RunContainerError: sandbox container \"1dc34867fe06c2dadbbe2f4a2f923f4ff947abadc52ed567e2a5f2240670a4ea\" is not running"
K3S: E0627 08:27:58.926296      55 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"init-permissions\" with RunContainerError: \"sandbox container \\\"1dc34867fe06c2dadbbe2f4a2f923f4ff947abadc52ed567e2a5f2240670a4ea\\\" is not running\"" pod="test/keycloak-deployment-769c677ffc-ps2f4" podUID="9b972221-934f-49b0-8e50-a4707f900998"
K3S: I0627 08:27:59.035466      55 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1dc34867fe06c2dadbbe2f4a2f923f4ff947abadc52ed567e2a5f2240670a4ea"
K3S: I0627 08:27:59.037239      55 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4279377afe1c0e859a25c2fc9afb94f26c70c93ca48b04876cdb4b469d2477a"
K3S: I0627 08:27:59.041369      55 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b66735821f867f505b38b4dbb33f3d5030696a478a2d7fc153c3e643e584888"
K3S: I0627 08:27:59.046860      55 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b02e5c9f92e9e7c2a3d67ae2fe549a36668fe3ffbfd5a1e767956c187b9eb43"
K3S: I0627 08:27:59.051189      55 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea35c69ffaef7cef01da890104a56a06aa11a282297473ed0846d1a5fe0cd4ce"
K3S: I0627 08:27:59.156116      55 scope.go:117] "RemoveContainer" containerID="eafd27bae0981a09d8f1be86f1a3224cfd79fffe3ef482ddd43784c0c12fce65"
K3S: E0627 08:27:59.289009      55 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"init-permissions\" with CrashLoopBackOff: \"back-off 10s restarting failed container=init-permissions pod=keycloak-deployment-769c677ffc-ps2f4_test(9b972221-934f-49b0-8e50-a4707f900998)\"" pod="test/keycloak-deployment-769c677ffc-ps2f4" podUID="9b972221-934f-49b0-8e50-a4707f900998"
K3S: E0627 08:27:59.358034      55 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"metrics-server\" with CrashLoopBackOff: \"back-off 20s restarting failed container=metrics-server pod=metrics-server-6f4c6675d5-s7pp2_kube-system(ec1dcbc4-20fa-4372-b85c-be5348d10a84)\"" pod="kube-system/metrics-server-6f4c6675d5-s7pp2" podUID="ec1dcbc4-20fa-4372-b85c-be5348d10a84"
K3S: E0627 08:27:59.411868      55 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"coredns\" with CrashLoopBackOff: \"back-off 20s restarting failed container=coredns pod=coredns-697968c856-92jdv_kube-system(6b220854-264f-4415-8c7d-d3ca42872cd3)\"" pod="kube-system/coredns-697968c856-92jdv" podUID="6b220854-264f-4415-8c7d-d3ca42872cd3"
K3S: E0627 08:27:59.422362      55 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"local-path-provisioner\" with CrashLoopBackOff: \"back-off 20s restarting failed container=local-path-provisioner pod=local-path-provisioner-774c6665dc-hc695_kube-system(3a982885-c4ad-416d-9fff-ac92189b24fd)\"" pod="kube-system/local-path-provisioner-774c6665dc-hc695" podUID="3a982885-c4ad-416d-9fff-ac92189b24fd"
Additional Information
wsl.exe -v
WSL version: 2.5.7.0
Kernel version: 6.6.87.1-1
WSLg version: 1.0.66
MSRDC version: 1.2.6074
Direct3D version: 1.611.1-81528511
DXCore version: 10.0.26100.1-240331-1435.ge-release
Windows version: 10.0.26100.4349

WSL distribution: Ubuntu 24.04

Windows: 11 Enterprise
Version: 24H2
OS build: 26100.4349
Experience: Windows Feature Experience Pack 1000.26100.107.0
All available Windows updates installed

Followed informations on this GitHub page regarding the cgroup problems on WSL systems: https://github.com/spurin/wsl-cgroupsv2

docker run -it --rm spurin/wsl-cgroupsv2:latest returns Success: cgroup type is cgroup2fs on my machine.

I was able to create a green test by changing the constructor contents of the K3sContainer.java and by removing its cgroup configuration:


    public K3sContainer(DockerImageName dockerImageName) {
        super(dockerImageName);
        dockerImageName.assertCompatibleWith(new DockerImageName[]{DockerImageName.parse("rancher/k3s")});
        this.addExposedPorts(new int[]{KUBE_SECURE_PORT, RANCHER_WEBHOOK_PORT});
        this.setPrivilegedMode(true);
        // removed the two lines below
        // this.withCreateContainerCmdModifier((it) -> it.getHostConfig().withCgroupnsMode("host"));
        // this.addFileSystemBind("/sys/fs/cgroup", "/sys/fs/cgroup", BindMode.READ_WRITE);
        Map<String, String> tmpFsMapping = new HashMap();
        tmpFsMapping.put("/run", "");
        tmpFsMapping.put("/var/run", "");
        this.setTmpFsMapping(tmpFsMapping);
        this.setCommand(new String[]{"server", "--disable=traefik", "--tls-san=" + this.getHost()});
        this.setWaitStrategy(Wait.forLogMessage(".*Node controller sync successful.*", 1));
    }

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The report names K3sContainer and the org.testcontainers:k3s dependency but no source file or test. Reproduce the startup failure on Windows 11 WSL2 with Rancher Desktop's Moby engine, then trace the cgroup-related container startup path. Done means the embedded K3s pods start successfully on the reported setup without regressing the working Mac behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, java, kubernetes
Domain
infrastructure, operating-systems, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.