Create kubeadm token for joining nodes with 24h expiration fail.

Organization Name:HIT

Short summary about the issue/question:

fatal: [master -> 10.10.8.87]: FAILED! => {"attempts": 5, "changed": true, "cmd": ["/usr/local/bin/kubeadm", "--kubeconfig", "/etc/kubernetes/admin.conf", "token", "create"], "delta": "0:01:15.017698", "end": "2021-11-04 14:19:20.079711", "msg": "non-zero return code", "rc": 1, "start": "2021-11-04 14:18:05.062013", "stderr": "timed out waiting for the condition", "stderr_lines": ["timed out waiting for the condition"], "stdout": "", "stdout_lines": []}

Brief what process you are following: follow #5592

• Operating type: Initial deployment

OpenPAI Environment:

• OpenPAI version: 1.8.0
• OS (e.g. from /etc/os-release): Ubuntu 18.04.6 LTS
• Kernel (e.g. uname -a):4.15.0-161-generic

TASK [kubernetes/master : kubeadm | Create kubeadm config] **********************************************************************************************************************************************************
Thursday 04 November 2021  22:11:22 +0800 (0:00:00.031)       0:02:29.371 ***** 
changed: [master]

TASK [kubernetes/master : Backup old certs and keys] ****************************************************************************************************************************************************************
Thursday 04 November 2021  22:11:22 +0800 (0:00:00.563)       0:02:29.934 ***** 
ok: [master] => (item={'src': 'apiserver.crt', 'dest': 'apiserver.crt.old'})
ok: [master] => (item={'src': 'apiserver.key', 'dest': 'apiserver.key.old'})
ok: [master] => (item={'src': 'apiserver-kubelet-client.crt', 'dest': 'apiserver-kubelet-client.crt.old'})
ok: [master] => (item={'src': 'apiserver-kubelet-client.key', 'dest': 'apiserver-kubelet-client.key.old'})
ok: [master] => (item={'src': 'front-proxy-client.crt', 'dest': 'front-proxy-client.crt.old'})
ok: [master] => (item={'src': 'front-proxy-client.key', 'dest': 'front-proxy-client.key.old'})

TASK [kubernetes/master : kubeadm | Initialize first master] ********************************************************************************************************************************************************
Thursday 04 November 2021  22:11:24 +0800 (0:00:01.101)       0:02:31.035 ***** 

TASK [kubernetes/master : set kubeadm certificate key] **************************************************************************************************************************************************************
Thursday 04 November 2021  22:11:24 +0800 (0:00:00.026)       0:02:31.062 ***** 

TASK [kubernetes/master : Create hardcoded kubeadm token for joining nodes with 24h expiration (if defined)] ********************************************************************************************************
Thursday 04 November 2021  22:11:24 +0800 (0:00:00.033)       0:02:31.095 ***** 

TASK [kubernetes/master : Create kubeadm token for joining nodes with 24h expiration (default)] *********************************************************************************************************************
Thursday 04 November 2021  22:11:24 +0800 (0:00:00.040)       0:02:31.135 ***** 
FAILED - RETRYING: Create kubeadm token for joining nodes with 24h expiration (default) (5 retries left).
FAILED - RETRYING: Create kubeadm token for joining nodes with 24h expiration (default) (4 retries left).
FAILED - RETRYING: Create kubeadm token for joining nodes with 24h expiration (default) (3 retries left).
FAILED - RETRYING: Create kubeadm token for joining nodes with 24h expiration (default) (2 retries left).
FAILED - RETRYING: Create kubeadm token for joining nodes with 24h expiration (default) (1 retries left).
fatal: [master -> 10.10.8.87]: FAILED! => {"attempts": 5, "changed": true, "cmd": ["/usr/local/bin/kubeadm", "--kubeconfig", "/etc/kubernetes/admin.conf", "token", "create"], "delta": "0:01:15.017698", "end": "2021-11-04 14:19:20.079711", "msg": "non-zero return code", "rc": 1, "start": "2021-11-04 14:18:05.062013", "stderr": "timed out waiting for the condition", "stderr_lines": ["timed out waiting for the condition"], "stdout": "", "stdout_lines": []}

NO MORE HOSTS LEFT **************************************************************************************************************************************************************************************************

PLAY RECAP **********************************************************************************************************************************************************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
master                     : ok=229  changed=23   unreachable=0    failed=1    skipped=339  rescued=0    ignored=1   
worker1                    : ok=147  changed=11   unreachable=0    failed=0    skipped=200  rescued=0    ignored=0   

Start with the Kubernetes master Ansible task named “Create kubeadm token for joining nodes with 24h expiration (default)” and inspect the kubeadm token command and its retry output. Compare the deployment process with issue #5592 and investigate why the command times out on the reported environment. Done means initial deployment creates the token successfully and proceeds past this task.