kubeslice / kubeslice/kubeslice-cli

Bug: `kubeslice-dns` and `kubeslice-operator` still pending while installation of kubeslice in worker clusters on gke standard tier due to pod affinity error

Open
#103 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Go
Stars
26
Forks
27
PR merge metrics
No merged PRs in 30d

Description

### 📜 Description

I tried setting up kubeslice via manual yaml instead of topology , I registered controller and worker clusters on my gke clusters , so I figured few things before creating slice . the `kubeslice-dns` and `kubeslice-operator` pods are keep on showing status as pending in `kubeslice-system` namespace in both of the worker clusters

### 👟 Reproduction steps

- Create 3 GKE Standard tier clusters on same or different regions
- Manage to have a topology file of cluster and workers with its kubeconfig
- Finally run the installation using kubeslice-cli

### 👍 Expected behavior

Kubeslice needed to be installed on the gke clusters

### 👎 Actual Behavior

Doesn't work at all

### 🐚 Relevant log output

These are the logs from one of the worker cluster while installation of kubeslice via helm upgrade
```shell
(base) ➜ yaml-cofing kubectl get pods --all-namespaces
Found existing alias for "kubectl get pods --all-namespaces". You should use: "kgpa"
NAMESPACE NAME READY STATUS RESTARTS AGE
gke-managed-cim kube-state-metrics-0 2/2 Running 0 42m
gmp-system collector-7c77w 2/2 Running 0 41m
gmp-system collector-jlmlm 2/2 Running 0 41m
gmp-system collector-lhg2q 2/2 Running 0 41m
gmp-system gmp-operator-5899b68d4b-kf9lc 1/1 Running 0 41m
kube-system event-exporter-gke-6d7c4dcf79-77zkz 2/2 Running 0 42m
kube-system fluentbit-gke-9hbcd 3/3 Running 0 41m
kube-system fluentbit-gke-snnm2 3/3 Running 0 41m
kube-system fluentbit-gke-w6gtm 3/3 Running 0 41m
kube-system gke-metrics-agent-sngx6 2/2 Running 0 41m
kube-system gke-metrics-agent-sxplv 2/2 Running 0 41m
kube-system gke-metrics-agent-ts9s4 2/2 Running 0 41m
kube-system konnectivity-agent-8596fd4d6f-gbk89 2/2 Running 0 40m
kube-system konnectivity-agent-8596fd4d6f-l4kbz 2/2 Running 0 41m
kube-system konnectivity-agent-8596fd4d6f-thjck 2/2 Running 0 40m
kube-system konnectivity-agent-autoscaler-57cb65694f-6v2t8 1/1 Running 0 41m
kube-system kube-dns-67c79cd964-jh57p 4/4 Running 0 40m
kube-system kube-dns-67c79cd964-pplkw 4/4 Running 0 42m
kube-system kube-dns-autoscaler-69778b8cfb-rwlft 1/1 Running 0 41m
kube-system kube-proxy-gke-ks-worker-1-default-pool-24d47686-zdxl 1/1 Running 0 41m
kube-system kube-proxy-gke-ks-worker-1-default-pool-7c236418-7bx6 1/1 Running 0 40m
kube-system kube-proxy-gke-ks-worker-1-default-pool-c269c09f-c4d0 1/1 Running 0 40m
kube-system l7-default-backend-dcfd7d6bb-xjcll 1/1 Running 0 41m
kube-system metrics-server-v1.33.0-5d6c8599c6-sb66g 1/1 Running 0 41m
kube-system pdcsi-node-fhsfn 2/2 Running 0 41m
kube-system pdcsi-node-qwltv 2/2 Running 0 41m
kube-system pdcsi-node-stq57 2/2 Running 0 41m
kubeslice-nsm-webhook-system nsm-admission-webhook-k8s-58b6d9bf6b-6265q 1/1 Running 0 25m
kubeslice-nsm-webhook-system nsm-admission-webhook-k8s-58b6d9bf6b-vhfkg 1/1 Running 0 25m
kubeslice-system forwarder-kernel-759g8 0/1 Pending 0 25m
kubeslice-system forwarder-kernel-frvck 1/1 Running 0 25m
kubeslice-system forwarder-kernel-lfg9f 1/1 Running 0 25m
kubeslice-system kubeslice-dns-79b7d7fbf4-25h6z 0/1 Pending 0 25m
kubeslice-system kubeslice-install-crds-4t47r 0/1 Completed 0 26m
kubeslice-system kubeslice-operator-5d9c956cd6-8ww2b 0/2 Pending 0 25m
kubeslice-system nsm-install-crds-gfhfh 0/1 Completed 0 25m
kubeslice-system nsmgr-4cb9x 2/2 Running 0 25m
kubeslice-system nsmgr-4j52t 2/2 Running 0 25m
kubeslice-system nsmgr-f9x5b 0/2 Pending 0 25m
kubeslice-system registry-k8s-979455d6d-8rkwc 1/1 Running 0 25m
kubeslice-system spire-install-clusterid-cr-9vgmn 0/1 Completed 0 25m
kubeslice-system spire-install-crds-2kjkd 0/1 Completed 0 25m
spire spiffe-csi-driver-bhxgt 2/2 Running 0 25m
spire spiffe-csi-driver-jz2jm 2/2 Running 0 25m
spire spiffe-csi-driver-pz6gm 2/2 Running 0 25m
spire spire-agent-66mtz 1/1 Running 0 25m
spire spire-agent-9txkb 1/1 Running 0 25m
spire spire-agent-rsn9g 1/1 Running 0 25m
spire spire-server-0 2/2 Running 0 25m

```

```shell

(base) ➜ yaml-cofing kubectl describe pods/kubeslice-dns-79b7d7fbf4-25h6z -n kubeslice-system
Found existing alias for "kubectl". You should use: "k"
Name: kubeslice-dns-79b7d7fbf4-25h6z
Namespace: kubeslice-system
Priority: 0
Service Account: kubeslice-dns
Node:
Labels: app=kubeslice-dns
kubeslice.io/pod-type=dns
pod-template-hash=79b7d7fbf4
Annotations: cloud.google.com/cluster_autoscaler_unhelpable_since: 2025-08-07T13:16:08+0000
cloud.google.com/cluster_autoscaler_unhelpable_until: Inf
Status: Pending
IP:
IPs:
Controlled By: ReplicaSet/kubeslice-dns-79b7d7fbf4
Containers:
dns:
Image: docker.io/aveshasystems/dns:0.1.4
Ports: 1053/UDP, 1053/TCP
Host Ports: 0/UDP, 0/TCP
Limits:
cpu: 50m
memory: 128Mi
Requests:
cpu: 10m
memory: 64Mi
Environment:
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-k54t8 (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
kube-api-access-k54t8:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: Burstable
Node-Selectors:
Tolerations: kubeslice.io/node-type=gateway:NoSchedule
kubeslice.io/node-type=gateway:NoExecute
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 30m default-scheduler 0/3 nodes are available: 3 node(s) didn't match Pod's node affinity/selector. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.
Warning FailedScheduling 20m (x2 over 25m) default-scheduler 0/3 nodes are available: 3 node(s) didn't match Pod's node affinity/selector. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.
Normal NotTriggerScaleUp 11s (x93 over 30m) cluster-autoscaler Pod didn't trigger scale-up:
(base) ➜ yaml-cofing kubectl describe pods/kubeslice-operator-5d9c956cd6-8ww2b -n kubeslice-system
Found existing alias for "kubectl". You should use: "k"
Name: kubeslice-operator-5d9c956cd6-8ww2b
Namespace: kubeslice-system
Priority: 0
Service Account: kubeslice-controller-manager
Node:
Labels: control-plane=controller-manager
pod-template-hash=5d9c956cd6
spoke-cluster=gke_graphic-transit-458312-f7_us-east4_ks-worker-1
Annotations: cloud.google.com/cluster_autoscaler_unhelpable_since: 2025-08-07T13:16:20+0000
cloud.google.com/cluster_autoscaler_unhelpable_until: Inf
kubectl.kubernetes.io/default-container: manager
prometheus.io/port: 8080
prometheus.io/scrape: true
Status: Pending
IP:
IPs:
Controlled By: ReplicaSet/kubeslice-operator-5d9c956cd6
Containers:
kube-rbac-proxy:
Image: gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0
Port: 8443/TCP
Host Port: 0/TCP
Args:
--secure-listen-address=0.0.0.0:8443
--upstream=http://127.0.0.1:8080/
--logtostderr=true
--v=10
Environment:
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-7xn8h (ro)
manager:
Image: docker.io/aveshasystems/worker-operator:1.4.0
Port:
Host Port:
Command:
/manager
Args:
--health-probe-bind-address=:8081
--metrics-bind-address=:8080
--leader-elect
Limits:
cpu: 500m
memory: 128Mi
Requests:
cpu: 10m
memory: 64Mi
Liveness: http-get http://:8081/healthz delay=15s timeout=1s period=20s #success=1 #failure=3
Readiness: http-get http://:8081/readyz delay=5s timeout=1s period=10s #success=1 #failure=3
Environment:
LOG_LEVEL: INFO
HUB_HOST_ENDPOINT: Optional: false
HUB_PROJECT_NAMESPACE: Optional: false
CLUSTER_NAME: gke_graphic-transit-458312-f7_us-east4_ks-worker-1
AVESHA_VL3_ROUTER_IMAGE: docker.io/aveshasystems/cmd-nse-vl3:1.0.6
AVESHA_VL3_ROUTER_PULLPOLICY: IfNotPresent
AVESHA_VL3_SIDECAR_IMAGE: docker.io/aveshasystems/kubeslice-router-sidecar:1.4.6
AVESHA_VL3_SIDECAR_IMAGE_PULLPOLICY: IfNotPresent
CLUSTER_ENDPOINT: https://34.86.91.118
AVESHA_GW_SIDECAR_IMAGE: docker.io/aveshasystems/gw-sidecar:1.0.3
AVESHA_GW_SIDECAR_IMAGE_PULLPOLICY: IfNotPresent
AVESHA_OPENVPN_SERVER_IMAGE: docker.io/aveshasystems/openvpn-server.alpine:1.0.4
AVESHA_OPENVPN_SERVER_PULLPOLICY: IfNotPresent
AVESHA_OPENVPN_CLIENT_IMAGE: docker.io/aveshasystems/openvpn-client.alpine:1.0.4
AVESHA_OPENVPN_CLIENT_PULLPOLICY: IfNotPresent
AVESHA_SLICE_GW_EDGE_IMAGE: aveshasystems/slicegw-edge:1.0.6
WORKER_INSTALLER_IMAGE: docker.io/aveshasystems/worker-installer:1.5.0
Mounts:
/etc/webhook/certs from webhook-certs (ro)
/var/run/secrets/kubernetes.io/hub-serviceaccount from hub-secret (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-7xn8h (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
kubeslice-worker-event-schema-conf:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: kubeslice-worker-event-schema-conf
Optional: false
webhook-certs:
Type: Secret (a volume populated by a Secret)
SecretName: kubeslice-admission-webhook-certs
Optional: false
hub-secret:
Type: Secret (a volume populated by a Secret)
SecretName: kubeslice-hub
Optional: false
kube-api-access-7xn8h:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: Burstable
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 31m default-scheduler 0/3 nodes are available: 3 node(s) didn't match Pod's node affinity/selector. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.
Warning FailedScheduling 20m (x2 over 25m) default-scheduler 0/3 nodes are available: 3 node(s) didn't match Pod's node affinity/selector. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.
Normal NotTriggerScaleUp 22s (x91 over 30m) cluster-autoscaler Pod didn't trigger scale-up:
```

### Version

None

### 🖥️ What operating system are you seeing the problem on?

_No response_

### ✅ Proposed Solution

for temporary fix I Labeled all nodes as gateway : `kubectl label nodes --all kubeslice.io/node-type=gateway --overwrite` on every worker cluster

### 👀 Have you spent some time to check if this issue has been raised before?

- [x] I checked and didn't find any similar issue

### Code of Conduct

- [x] I agree to follow this project's Code of Conduct

Contributor guide

Open the contributing guide

Research direction

Start with the kubeslice-cli installation path that runs the Helm upgrade, then reproduce the issue on the three-cluster GKE Standard setup described. Inspect the kubeslice-dns and kubeslice-operator pod descriptions and scheduler events for their node affinity requirements. Done means both pods schedule successfully and Kubeslice installs on the worker clusters.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, go, helm, kubernetes
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.