pingcap / pingcap/tidb-operator

Tikv Pod Not Starting

Open
#4,528 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
1.3k
Forks
540
Avg merge
3d 2h
Merged PRs (30d)
18

Description

Bug Report

What version of Kubernetes are you using?
1.22.1 (Container Runtime = 1.4.9 e25210fe30a0a703442421b0f60afac609f950a3)

What version of TiDB Operator are you using?
TiDB Operator Version: version.Info{GitVersion:"v1.3.3", GitCommit:"8949b33a38396c417331859fa00c670d1a627388", GitTreeState:"clean", BuildDate:"2022-05-17T02:47:18Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}

What's the status of the TiDB cluster pods?

NAME                                       READY   STATUS             RESTARTS      AGE    IP              NODE                NOMINATED NODE   READINESS GATES
advanced-tidb-discovery-6d6454b5c6-gtp7c   1/1     Running            0             13m    10.244.75.188   k8stesty1worker01   <none>           <none>
advanced-tidb-pd-0                         1/1     Running            0             13m    10.244.39.216   k8stesty1worker04   <none>           <none>
advanced-tidb-pd-1                         1/1     Running            0             13m    10.244.39.248   k8stesty1worker04   <none>           <none>
advanced-tidb-pd-2                         1/1     Running            0             13m    10.244.75.148   k8stesty1worker01   <none>           <none>
advanced-tidb-tikv-0                       0/1     CrashLoopBackOff   3 (30s ago)   74s    10.244.40.14    k8stesty1worker04   <none>           <none>
advanced-tidb-tikv-1                       0/1     CrashLoopBackOff   3 (20s ago)   68s    10.244.75.130   k8stesty1worker01   <none>           <none>
advanced-tidb-tikv-2                       0/1     Error              4 (54s ago)   101s   10.244.219.89   k8stesty1worker06   <none>           <none>

What did you see instead?

/ # RUST_BACKTRACE=full /tikv-server --pd=http://advanced-tidb-pd:2379 --advertise-addr=advanced-tidb-tikv-2.advanced-tidb-tikv-peer.tidb-cluster.svc:20160 --addr=0.0.0.0:20160 --status-addr=0.0.0.0:20180 --advertise-status-addr=advanced-tidb-tikv-2.advanced-tidb-tikv-peer.tidb-cluster.svc:20180 --data-dir=/var/lib/tikv --capacity=0 --config=/etc/tikv/tikv.toml

thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', components/tikv_util/src/sys/cgroup.rs:224:44
stack backtrace:
   0:     0x55650510b2b3 - std::backtrace_rs::backtrace::libunwind::trace::h99dbb39dca18857d at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/std/src/../../backtrace/src/backtrace/libunwind.rs:90:5
   1:     0x55650510b2b3 - std::backtrace_rs::backtrace::trace_unsynchronized::h832861927e9cfedf at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x55650510b2b3 - std::sys_common::backtrace::_print_fmt::h3d18154c77dcf310 at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/std/src/sys_common/backtrace.rs:67:5
   3:     0x55650510b2b3 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::he312f4ad5b9bb346
at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/std/src/sys_common/backtrace.rs:46:22
   4:     0x556503f7e4cc - core::fmt::write::h9a6d9c74526a6c1b
                               at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/core/src/fmt/mod.rs:1115:17
   5:     0x5565051095f4 - std::io::Write::write_fmt::h6aced00850e8186f
                               at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/std/src/io/mod.rs:1665:15
   6:     0x55650510a36b - std::sys_common::backtrace::_print::h65d996766de40da4
                               at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/std/src/sys_common/backtrace.rs:49:5
   7:     0x55650510a36b - std::sys_common::backtrace::print::h40df9727e635f303
                               at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/std/src/sys_common/backtrace.rs:36:9
   8:     0x55650510a36b - std::panicking::default_hook::{{closure}}::hd2da4327dea91a51
                               at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/std/src/panicking.rs:208:50
   9:     0x556505109285 - std::panicking::default_hook::h3d55120ad6ada158
                               at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/std/src/panicking.rs:225:9
  10:     0x556505109285 - std::panicking::rust_panic_with_hook::hf85dd0bb545e3b55
                               at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/std/src/panicking.rs:622:17
  11:     0x5565051231be - std::panicking::begin_panic_handler::{{closure}}::h736ae969434da9fa
                               at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/std/src/panicking.rs:517:13
  12:     0x55650512315c - std::sys_common::backtrace::__rust_end_short_backtrace::h6133bb80b1d6c3e0
                               at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/std/src/sys_common/backtrace.rs:141:18
  13:     0x55650512310d - rust_begin_unwind
                               at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/std/src/panicking.rs:515:5
  14:     0x556503cb1c80 - core::panicking::panic_fmt::hcf5f6d96e1dd7099
                               at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/core/src/panicking.rs:92:14
  15:     0x556503cb1ddc - core::panicking::panic::hd695e3b1d0dd4ef4
                               at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/core/src/panicking.rs:50:5
  16:     0x5565058899b9 - core::option::Option<T>::unwrap::hdd289ad090176f45
                               at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/core/src/option.rs:722:21
  17:     0x5565058899b9 - tikv_util::sys::cgroup::build_path::hf107d7b765027ab3
                               at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/tikv_util/src/sys/cgroup.rs:224:20
  18:     0x556505887e12 - tikv_util::sys::cgroup::CGroupSys::cpuset_cores::hf52b8ad58452123d
                               at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/tikv_util/src/sys/cgroup.rs:102:28
  19:     0x5565058878bd - tikv_util::sys::SysQuota::cpu_cores_quota::h7c9eb6cd7e8628e4
                               at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/tikv_util/src/sys/mod.rs:35:28
  20:     0x556505471a46 - <tikv::config::UnifiedReadPoolConfig as core::default::Default>::default::h74726c47e6e71835
                               at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/config.rs:1730:23
  21:     0x556505471a46 - <tikv::config::ReadPoolConfig as core::default::Default>::default::h57dd7aa86d9ddc8f
                               at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/config.rs:2033:5
  22:     0x556505471a46 - <tikv::config::TiKvConfig as core::default::Default>::default::h48dbc32208f0862a
                               at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/config.rs:2563:23
  23:     0x5565054e3aef - <tikv::config::_IMPL_DESERIALIZE_FOR_TiKvConfig::<impl serde::de::Deserialize for tikv::config::TiKvConfig>::deserialize::__Visitor as serde::de::Visitor>::visit_map::h342d0215db819a89
                               at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/config.rs:2431:28
  24:     0x556505482752 - <&mut toml::de::Deserializer as serde::de::Deserializer>::deserialize_any::haacb93ba568c4845
                               at /rust/registry/src/github.com-1ecc6299db9ec823/toml-0.5.7/src/de.rs:220:19
  25:     0x556505482752 - <&mut toml::de::Deserializer as serde::de::Deserializer>::deserialize_struct::h9b968e2c55d0a6fd
                               at /rust/registry/src/github.com-1ecc6299db9ec823/toml-0.5.7/src/de.rs:315:9
  26:     0x556505482752 - tikv::config::_IMPL_DESERIALIZE_FOR_TiKvConfig::<impl serde::de::Deserialize for tikv::config::TiKvConfig>::deserialize::h10801840d7967723
                               at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/config.rs:2431:28
  27:     0x556505482752 - tikv::config::TiKvConfig::from_file::h88010f8442c80eda
                               at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/config.rs:3066:13
  28:     0x556503dc090f - tikv_server::main::{{closure}}::hd68fcc929c06121f
                               at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/cmd/tikv-server/src/main.rs:163:13
  29:     0x556503dc090f - core::option::Option<T>::map_or_else::haf15498ea772a261
                               at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/core/src/option.rs:873:24
  30:     0x556503dc090f - tikv_server::main::h9917e319c97ac794
                               at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/cmd/tikv-server/src/main.rs:159:22
  31:     0x556503da7cf3 - core::ops::function::FnOnce::call_once::h52dabdb5f71dcd1a
                               at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/core/src/ops/function.rs:227:5
  32:     0x556503da7cf3 - std::sys_common::backtrace::__rust_begin_short_backtrace::hc16968655169a599
                               at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/std/src/sys_common/backtrace.rs:125:18
  33:     0x556503dc4354 - main
  34:     0x7f0baaa18d9b - __libc_start_main
  35:     0x556503da7287 - <unknown>
  36:                0x0 - <unknown>

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at components/tikv_util/src/sys/cgroup.rs:224, then follow the cpuset_cores and cpu_cores_quota callers shown in the backtrace through tikv/src/config.rs. Reproduce with the displayed tikv-server command and inspect the container's cgroup configuration; done means the same Kubernetes TiKV pod no longer panics during startup.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, rust
Domain
databases, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.