unable to start TiKV due to DNS resolution

Bug Report

What version of Kubernetes are you using?

1.27.3 under k3s distribution

What version of TiDB Operator are you using?

1.4.5

What storage classes exist in the Kubernetes cluster and what are used for PD/TiKV pods?

local-path

What's the status of the TiDB cluster pods?

Running

What did you do?

What did you expect to see?

TiKV successfully started and able to connect to PD server

What did you see instead?

TiKV unable to start with these following logs. Altho running curl directly within TiKV pod successfully, TiKV unable to connect to PD server

[2023/07/05 04:21:36.755 +00:00] [INFO] [util.rs:598] ["connecting to PD endpoint"] [endpoints=http://tidb-pd:2379]
[2023/07/05 04:21:38.756 +00:00] [INFO] [util.rs:560] ["PD failed to respond"] [err="Grpc(RpcFailure(RpcStatus { code: 4-DEADLINE_EXCEEDED, message: \"Deadline Exceeded\", details: [] }))"] [endpoints=http://tidb-pd:2379]
[2023/07/05 04:21:39.057 +00:00] [INFO] [util.rs:598] ["connecting to PD endpoint"] [endpoints=http://tidb-pd:2379]
[2023/07/05 04:21:41.058 +00:00] [INFO] [util.rs:560] ["PD failed to respond"] [err="Grpc(RpcFailure(RpcStatus { code: 4-DEADLINE_EXCEEDED, message: \"Deadline Exceeded\", details: [] }))"] [endpoints=http://tidb-pd:2379]
[2023/07/05 04:21:41.359 +00:00] [INFO] [util.rs:598] ["connecting to PD endpoint"] [endpoints=http://tidb-pd:2379]

My hypothesis is that somehow single level DNS was unable to resolve. I tried to edit TiKV configmap to change it to ${CLUSTER_NAME}-pd.${NAMESPACE}.svc:2379 then it was successfully connect but later got reversed by the operator

Propose fix at https://github.com/pingcap/tidb-operator/pull/5145

Start by tracing how the TiKV configmap and PD endpoint are generated in the TiDB Operator, using the reported util.rs connection logs and the ${CLUSTER_NAME}-pd.${NAMESPACE}.svc:2379 workaround as reference. Review pull request 5145, then verify that the operator retains the working DNS form and TiKV connects to PD without the configmap change being reverted.