pingcap / pingcap/tiup

`tiup cluster start <cluster_name>` output shows success, but no TiKV process runs on the designated nodes

Open
#2,204 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

type/bug
Dominant language
Go
Stars
466
Forks
338
Avg merge
3d 7h
Merged PRs (30d)
8

Description

Bug Report

Please answer these questions before submitting your issue. Thanks!

  1. What did you do?

Step 1 - Use the following topology.yaml:

global:
  user: "tikv"
  ssh_port: 22
  deploy_dir: "/tikv-deploy"
  data_dir: "/tikv-data"
pd_servers:
  - host: 192.168.151.250
  - host: 192.168.133.47
  - host: 192.168.172.40
tikv_servers:
  - host: 192.168.173.212
  - host: 192.168.131.50
  - host: 192.168.131.101
monitoring_servers:
  - host: 192.168.202.125
grafana_servers:
  - host: 192.168.202.125

Step 2 - Run tiup cluster check ./topology.yaml --apply --user root and observe that one Fail persists: service irqbalance is not running.

Step 3 - Run tiup cluster deploy tikv v6.5.2 ./topology.yaml --user root, followed by tiup cluster start tikv. Observe the following in the output:

Starting component tikv
	Starting instance 192.168.131.101:20160
	Starting instance 192.168.173.212:20160
	Starting instance 192.168.131.50:20160
	Start instance 192.168.131.50:20160 success
	Start instance 192.168.173.212:20160 success
	Start instance 192.168.131.101:20160 success
  1. What did you expect to see?

When running tiup cluster display tikv, we expect to see:

192.168.131.101:20160  tikv        192.168.131.101  20160/20180  linux/x86_64  Up    /tikv-data/tikv-20160       /tikv-deploy/tikv-20160
192.168.131.50:20160   tikv        192.168.131.50   20160/20180  linux/x86_64  Up    /tikv-data/tikv-20160       /tikv-deploy/tikv-20160
192.168.173.212:20160  tikv        192.168.173.212  20160/20180  linux/x86_64  Up    /tikv-data/tikv-20160       /tikv-deploy/tikv-20160

and in the nodes themselves, expect to see non-empty output upon running sudo lsof -i:20160.

  1. What did you see instead?
192.168.131.101:20160  tikv        192.168.131.101  20160/20180  linux/x86_64  Down    /tikv-data/tikv-20160       /tikv-deploy/tikv-20160
192.168.131.50:20160   tikv        192.168.131.50   20160/20180  linux/x86_64  Down    /tikv-data/tikv-20160       /tikv-deploy/tikv-20160
192.168.173.212:20160  tikv        192.168.173.212  20160/20180  linux/x86_64  Down    /tikv-data/tikv-20160       /tikv-deploy/tikv-20160

and in the nodes themselves, an empty output upon running sudo lsof -i:20160.

  1. What version of TiUP are you using (tiup --version)?

1.12.1

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the tiup cluster check, deploy, start, and display command flow described in the report, reproducing it with the supplied topology.yaml and TiUP 1.12.1. Compare the start output with tiup cluster display and sudo lsof -i:20160 on each designated node; done means failed starts are detected rather than reported as successful, and process status matches the command output.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cli, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.