Tidb pods failed on a specific host

What happened?

On a Kubernetes v1.25.4 cluster, the tidb pods scheduled by the helm chart tidb-cluster v1.3.9 fails over and over and over again on a specific host.

The cluster consists of 2 hosts (control-plane included), initially, the pd and tikv pods (all configured 1 replica) are all ok, and are scheduled respectively to different hosts.
The tidb pods on the same host with the tikv are successfully up and running, but the other 2 tidb pods on the same host with the pd however fail again and again.
The logs on those pods is as follows:

start tidb-server ...
/tidb-server --store=tikv --advertise-address=tidb-cluster-v1-3-9-tidb-1.tidb-cluster-v1-3-9-tidb-peer.the-black-pearl.svc --host=0.0.0.0 --path=tidb-cluster-v1-3-9-pd:2379 --config=/etc/tidb/tidb.toml
 --log-slow-query=/var/log/tidb/slowlog
[2022/12/13 01:28:25.427 +00:00] [INFO] [cpuprofile.go:115] ["parallel cpu profiler started"]
[2022/12/13 01:28:25.427 +00:00] [FATAL] [terror.go:292] ["unexpected error"] [error="open /sys/fs/cgroup/cpu,cpuacct/system.slice/containerd.service/kubepods-burstable-pod44b8861b_6533_4eb4_8ced_54541a2ee207.slice:cri-containerd:55e94844d492ba2a33bbc69305559911da2426636a86b6041b801fdddb490209/cpu.cfs_quota_us: no such file or directory"] [stack="github.com/pingcap/tidb/parser/terror.MustNil\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:292\nmain.setGlobalVars\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/tidb-server/main.go:589\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/tidb-server/main.go:195\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"] [stack="github.com/pingcap/tidb/parser/terror.MustNil\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/parser/terror/terror.go:292\nmain.setGlobalVars\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/tidb-server/main.go:589\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/tidb-server/main.go:195\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"]

And as what I've inspected, the /sys/fs/cgroup/cpu,cpuacct/system.slice/containerd.service/kubepods-burstable-pod44b8861b_6533_4eb4_8ced_54541a2ee207.slice:cri-containerd:55e94844d492ba2a33bbc69305559911da2426636a86b6041b801fdddb490209/cpu.cfs_quota_us indeed does not exist, but I can not figure out the cause.

There might be some host system misconfiguration or something... hoping for some guidances

Additional information

I found that there are some other 2 folders starting with kubepods-burstable-pod44b8861b_6533_4eb4_8ced_54541a2ee207.slice:cri-containerd: inside the folder /sys/fs/cgroup/cpu,cpuacct/system.slice/containerd.service/ which all have valid cpu.cfs_quota_us files. The valid folders are but with different hash parts

Containerd and Kubernetes version

$ containerd --version
containerd containerd.io 1.5.11 3df54a852345ae127d1fa3092b95168e4a88e2f8

$ kubectl version --output=yaml
clientVersion:
  buildDate: "2022-11-09T13:36:36Z"
  compiler: gc
  gitCommit: 872a965c6c6526caa949f0c6ac028ef7aff3fb78
  gitTreeState: clean
  gitVersion: v1.25.4
  goVersion: go1.19.3
  major: "1"
  minor: "25"
  platform: linux/amd64
kustomizeVersion: v4.5.7
serverVersion:
  buildDate: "2022-11-09T13:29:58Z"
  compiler: gc
  gitCommit: 872a965c6c6526caa949f0c6ac028ef7aff3fb78
  gitTreeState: clean
  gitVersion: v1.25.4
  goVersion: go1.19.3
  major: "1"
  minor: "25"
  platform: linux/amd64

OS version

$ cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

$ uname -a
Linux fallback-server 3.10.0-1160.31.1.el7.x86_64 #1 SMP Thu Jun 10 13:32:12 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

The said 2 hosts have identical outputs for above commands

The failure points to tidb-server/main.go at setGlobalVars, line 589, and the missing cpu.cfs_quota_us path is shown in the logs. Start by comparing the cgroup paths and container runtime details on both hosts, then trace how this entry point reads the quota file. Done means identifying the host or runtime condition that causes the missing path and documenting a reproducible fix or workaround.