TiKV keep connecting to pd with 4-DEADLINE_EXCEEDED error

Bug Report

What version of Kubernetes are you using?

Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.4", GitCommit:"b695d79d4f967c403a96986f1750a35eb75e75f1", GitTreeState:"clean", BuildDate:"2021-11-17T15:48:33Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"clean", BuildDate:"2021-05-21T23:01:33Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"}

What version of TiDB Operator are you using?

TiDB Operator Version: version.Info{GitVersion:"v1.3.3", GitCommit:"8949b33a38396c417331859fa00c670d1a627388", GitTreeState:"clean", BuildDate:"2022-05-17T02:47:18Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}

What storage classes exist in the Kubernetes cluster and what are used for PD/TiKV pods?

NAME                 PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
standard (default)   rancher.io/local-path   Delete          WaitForFirstConsumer   false                  95d

NAME                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
pd-chaos-pd-0       Bound    pvc-3b2fa30e-7c21-4411-b893-299d40912613   10Gi       RWO            standard       21m
pd-chaos-pd-1       Bound    pvc-b921799f-57d8-4998-8287-dcd0c10329cf   10Gi       RWO            standard       21m
pd-chaos-pd-2       Bound    pvc-2e51d966-07b4-4ac5-afe2-22d58d77c0fc   10Gi       RWO            standard       21m
tikv-chaos-tikv-0   Bound    pvc-347784a9-7455-4ae3-be88-dea53badaadb   100Gi      RWO            standard       20m
tikv-chaos-tikv-1   Bound    pvc-279d9124-72a7-49cf-bf64-aee1ed8419e0   100Gi      RWO            standard       20m
tikv-chaos-tikv-2   Bound    pvc-44af264d-6ec4-4a75-95c5-cb44e63bd666   100Gi      RWO            standard       20m

What's the status of the TiDB cluster pods?

NAME                               READY   STATUS    RESTARTS   AGE     IP            NODE                 NOMINATED NODE   READINESS GATES
chaos-discovery-7544f577cb-67g2t   1/1     Running   0          22m     10.244.0.73   kind-control-plane   <none>           <none>
chaos-pd-0                         1/1     Running   1          9m45s   10.244.0.91   kind-control-plane   <none>           <none>
chaos-pd-1                         1/1     Running   1          11m     10.244.0.90   kind-control-plane   <none>           <none>
chaos-pd-2                         1/1     Running   1          12m     10.244.0.89   kind-control-plane   <none>           <none>
chaos-tidb-0                       2/2     Running   0          20m     10.244.0.88   kind-control-plane   <none>           <none>
chaos-tidb-1                       2/2     Running   0          20m     10.244.0.86   kind-control-plane   <none>           <none>
chaos-tidb-2                       2/2     Running   0          20m     10.244.0.87   kind-control-plane   <none>           <none>
chaos-tikv-0                       1/1     Running   0          21m     10.244.0.84   kind-control-plane   <none>           <none>
chaos-tikv-1                       1/1     Running   0          21m     10.244.0.85   kind-control-plane   <none>           <none>
chaos-tikv-2                       1/1     Running   0          9m5s    10.244.0.92   kind-control-plane   <none>           <none>

What did you do?

I try to deploy by this instruction for tidb v5.4.1: https://docs.pingcap.com/tidb-in-kubernetes/stable/deploy-tidb-operator

but TIKV can not started, the error on TiKV pod was:

[2022/06/08 09:19:49.098 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=http://chaos-pd:2379]
[2022/06/08 09:19:51.099 +00:00] [INFO] [util.rs:506] ["PD failed to respond"] [err="Grpc(RpcFailure(RpcStatus { code: 4-DEADLINE_EXCEEDED, message: \"Deadline Exceeded\", details: [] }))"] [endpoints=http://chaos-pd:2379]
[2022/06/08 09:19:51.400 +00:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=http://chaos-pd:2379]
[2022/06/08 09:19:53.401 +00:00] [INFO] [util.rs:506] ["PD failed to respond"] [err="Grpc(RpcFailure(RpcStatus { code: 4-DEADLINE_EXCEEDED, message: \"Deadline Exceeded\", details: [] }))"] [endpoints=http://chaos-pd:2379]
[2022/06/08 09:19:53.401 +00:00] [WARN] [client.rs:153] ["validate PD endpoints failed"] [err="Other(\"[components/pd_client/src/util.rs:539]: PD cluster failed to respond\")"]

I have precheck all the prerequisites, destroy and retry to deploy but still got the same error.
Then I deploy tidb v3.0.12 and it started successfully, i try to upgrade it to v5.4.1, but guess what? The same error occurs.

What did you expect to see?
The cluster should be started successfully when following the instruction.

What did you see instead?
Error on TiKV when trying to connect to PD server.

Start with the TiDB Operator deployment instruction linked in the report, then reproduce the v5.4.1 deployment and inspect the TiKV and PD pod logs and Kubernetes service connectivity. Done means the cluster starts successfully without repeated 4-DEADLINE_EXCEEDED errors when following the documented deployment steps.