microsoft / microsoft/pai

pai node : Failed to connect to apiserver

Open
#5,646 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
JavaScript
Stars
2.7k
Forks
554
Avg merge
20h 42m
Merged PRs (30d)
14

Description

on pai node , run:systemctl status kubelet

● kubelet.service - Kubernetes Kubelet Server
Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2021-11-04 16:02:04 CST; 7s ago
Docs: https://github.com/GoogleCloudPlatform/kubernetes
Main PID: 22413 (kubelet)
Tasks: 26 (limit: 4915)
CGroup: /system.slice/kubelet.service
└─22413 /usr/local/bin/kubelet --logtostderr=true --v=2 --node-ip=192.168.50.191 --hostname-override=pai-worker1 --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --config=/etc/kubernet

11月 04 16:02:04 pai-worker1 kubelet[22413]: I1104 16:02:04.495889 22413 feature_gate.go:216] feature gates: &{map[]}
11月 04 16:02:04 pai-worker1 kubelet[22413]: I1104 16:02:04.496000 22413 plugins.go:103] No cloud provider specified.
11月 04 16:02:04 pai-worker1 kubelet[22413]: I1104 16:02:04.496029 22413 server.go:541] No cloud provider specified: "" from the config file: ""
11月 04 16:02:04 pai-worker1 kubelet[22413]: E1104 16:02:04.497419 22413 bootstrap.go:238] Unable to read existing bootstrap client config: invalid configuration: no configuration has been provided
11月 04 16:02:04 pai-worker1 kubelet[22413]: I1104 16:02:04.497444 22413 bootstrap.go:117] Using bootstrap kubeconfig to generate TLS client cert, key and kubeconfig file
11月 04 16:02:04 pai-worker1 kubelet[22413]: I1104 16:02:04.499277 22413 certificate_store.go:129] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-current.pem".
11月 04 16:02:04 pai-worker1 kubelet[22413]: I1104 16:02:04.521611 22413 bootstrap.go:293] Failed to connect to apiserver: Get https://192.168.50.190:6443/healthz?timeout=1s: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")
11月 04 16:02:06 pai-worker1 kubelet[22413]: I1104 16:02:06.534133 22413 bootstrap.go:293] Failed to connect to apiserver: Get https://192.168.50.190:6443/healthz?timeout=1s: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")
11月 04 16:02:08 pai-worker1 kubelet[22413]: I1104 16:02:08.827977 22413 bootstrap.go:293] Failed to connect to apiserver: Get https://192.168.50.190:6443/healthz?timeout=1s: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")

on master node:
root@pai-master:/home/fxkj# kubectl get node
NAME STATUS ROLES AGE VERSION
pai-master Ready master 25d v1.15.11
pai-worker1 NotReady 25d v1.15.11

kubectl describe nodes pai-worker1
Name: pai-worker1
Roles:
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=pai-worker1
kubernetes.io/os=linux
pai-worker=true
Annotations: kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Sat, 09 Oct 2021 16:54:41 +0800
Taints: node.kubernetes.io/unreachable:NoSchedule
Unschedulable: false
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message


NetworkUnavailable False Sat, 09 Oct 2021 17:07:06 +0800 Sat, 09 Oct 2021 17:07:06 +0800 CalicoIsUp Calico is running on this node
MemoryPressure Unknown Tue, 12 Oct 2021 15:39:09 +0800 Tue, 12 Oct 2021 19:34:06 +0800 NodeStatusUnknown Kubelet stopped posting node status.
DiskPressure Unknown Tue, 12 Oct 2021 15:39:09 +0800 Tue, 12 Oct 2021 19:34:06 +0800 NodeStatusUnknown Kubelet stopped posting node status.
PIDPressure Unknown Tue, 12 Oct 2021 15:39:09 +0800 Tue, 12 Oct 2021 19:34:06 +0800 NodeStatusUnknown Kubelet stopped posting node status.
Ready Unknown Tue, 12 Oct 2021 15:39:09 +0800 Tue, 12 Oct 2021 19:34:06 +0800 NodeStatusUnknown Kubelet stopped posting node status.
Addresses:
InternalIP: 192.168.50.191
Hostname: pai-worker1
Capacity:
cpu: 20
ephemeral-storage: 490691512Ki
github.com/fuse: 110
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 65525284Ki
nvidia.com/gpu: 4
pods: 110
Allocatable:
cpu: 19900m
ephemeral-storage: 452221296711
github.com/fuse: 110
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 65172884Ki
nvidia.com/gpu: 4
pods: 110
System Info:
Machine ID: 9f90e16ee38c4434aaa145f48b54e4f1
System UUID: 478f53c6-1797-cf67-ba8e-04d4c45b392c
Boot ID: 33833536-aa28-4ffa-9e26-4c8166727767
Kernel Version: 5.4.0-42-generic
OS Image: Ubuntu 18.04.5 LTS
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://18.9.7
Kubelet Version: v1.15.11
Kube-Proxy Version: v1.15.11
PodCIDR: 10.200.0.128/25
Non-terminated Pods: (10 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE


default job-exporter-xmtd4 0 (0%) 0 (0%) 0 (0%) 0 (0%) 23d
default log-manager-ds-tm2sj 0 (0%) 0 (0%) 0 (0%) 0 (0%) 23d
default node-exporter-7kbhz 0 (0%) 0 (0%) 0 (0%) 0 (0%) 23d
kube-system calico-kube-controllers-9f59f7848-bh9q4 30m (0%) 100m (0%) 64M (0%) 256M (0%) 22d
kube-system calico-node-27b29 150m (0%) 300m (1%) 64M (0%) 500M (0%) 21d
kube-system coredns-67db989964-m2mnj 0 (0%) 0 (0%) 0 (0%) 1000Mi (1%) 22d
kube-system dns-autoscaler-77486c4479-2f9th 20m (0%) 0 (0%) 10Mi (0%) 0 (0%) 22d
kube-system k8s-host-device-plugin-daemonset-bdx55 0 (0%) 0 (0%) 0 (0%) 0 (0%) 22d
kube-system kube-proxy-wjtrm 0 (0%) 0 (0%) 0 (0%) 0 (0%) 21d
kube-system nvidia-device-plugin-daemonset-hfkt5 0 (0%) 0 (0%) 0 (0%) 0 (0%) 22d
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits


cpu 200m (1%) 400m (2%)
memory 138485760 (0%) 1804576k (2%)
ephemeral-storage 0 (0%) 0 (0%)
github.com/fuse 0 0
nvidia.com/gpu 0 0
Events:

Available GPU nodes is 0;

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the reported systemctl status kubelet output and the Kubernetes commands kubectl get node and kubectl describe nodes pai-worker1. Trace the kubelet connection to the apiserver and certificate configuration; the issue currently provides no defined code change or acceptance condition, so completion would require reproducing the failure and documenting a verified resolution.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, linux
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.