pingcap / pingcap/tiup

TiKV is not active, but status is still UP

Open
#1,264 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

type/bug
Dominant language
Go
Stars
466
Forks
338
Avg merge
3d 7h
Merged PRs (30d)
8

Description

Bug Report

Please answer these questions before submitting your issue. Thanks!

  1. What did you do?
  1. Deploy a cluster with TLS enabled;

  2. tiup cluster reload the cluster

  3. Expect the reload should be successful, but it failed with below msg:
    image

  4. Seems TiKV is not upgrade successful, the log as below:
    tikv.log

  5. tiup cluster display the cluster

  6. What did you expect to see?

Expect the TiKV's status is Down or Inactive.

  1. What did you see instead?

image

Currently, we check TiKV's status via PD's store info https://github.com/pingcap/tiup/blob/bc4e43b5b793406fe36e81dc4b1fbec7f895e858/pkg/cluster/spec/tikv.go#L65-L79 and PD's store is actively reported by TiKV, if TiKV is down, the store maybe not GC in time.

BTW, after manually systemctl stop tikv-20160, the display status is correct, seems PD's store was updated:

image

  1. What version of TiUP are you using (tiup --version)?
the master branch

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with pkg/cluster/spec/tikv.go at the linked status-checking code, then reproduce the sequence with TLS, tiup cluster reload, and tiup cluster display. Use the linked TiKV log and PD store behavior to determine how an inactive TiKV should be represented; done means the display reports Down or Inactive instead of UP.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cli, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.