tikv / tikv/pd

leadership: unnecessary leadership election after pd-leader-io-hang

Open
#6,977 0 comments 0 reactions 1 assignee Claimed by @lhy1024 View on GitHub
type/enhancement
Dominant language
Go
Stars
1.2k
Forks
783
Avg merge
5d 21h
Merged PRs (30d)
36

Description

## Enhancement Task

when pd leader is injected pd-leader-io-hang, there is election for new leader, which is expected,

but when inject is finished after 5 minutes, there is new delete event without election, which is unexpected

```

2023-08-23T15:41:35+08:00 {"container":"pd","level":"INFO","namespace":"endless-ha-test-tps-2101058-1-152","log":"[raft.go:771] [\"3da436621ab53dd7 became leader at term 3\"]","pod":"tc-pd-1"}
2023-08-23T15:41:45+08:00 {"container":"pd","level":"INFO","namespace":"endless-ha-test-tps-2101058-1-152","log":"[leadership.go:310] [\"current leadership is deleted\"] [revision=1533] [leader-key=/pd/7270412980872193982/leader] [purpose=\"leader election\"]","pod":"tc-pd-1"}
2023-08-23T15:41:51+08:00 {"namespace":"endless-ha-test-tps-2101058-1-152","level":"INFO","pod":"tc-pd-0","container":"pd","log":"[leadership.go:310] [\"current leadership is deleted\"] [revision=1539] [leader-key=/pd/7270412980872193982/leader] [purpose=\"leader election\"]"}
2023-08-23T15:46:30+08:00 {"container":"pd","level":"INFO","namespace":"endless-ha-test-tps-2101058-1-152","log":"[leadership.go:310] [\"current leadership is deleted\"] [revision=1749] [leader-key=/pd/7270412980872193982/leader] [purpose=\"leader election\"]","pod":"tc-pd-1"}
2023-08-23T15:46:30+08:00 {"namespace":"endless-ha-test-tps-2101058-1-152","level":"INFO","pod":"tc-pd-0","container":"pd","log":"[leadership.go:310] [\"current leadership is deleted\"] [revision=1749] [leader-key=/pd/7270412980872193982/leader] [purpose=\"leader election\"]"}

2023-08-23T16:07:27+08:00 {"level":"INFO","container":"pd","pod":"tc-pd-2","namespace":"endless-ha-test-tps-2101058-1-152","log":"[raft.go:771] [\"16e8101bc52808d7 became leader at term 4\"]"}
2023-08-23T16:07:37+08:00 {"level":"INFO","container":"pd","pod":"tc-pd-2","namespace":"endless-ha-test-tps-2101058-1-152","log":"[leadership.go:310] [\"current leadership is deleted\"] [revision=2710] [leader-key=/pd/7270412980872193982/leader] [purpose=\"leader election\"]"}
2023-08-23T16:07:39+08:00 {"namespace":"endless-ha-test-tps-2101058-1-152","level":"INFO","pod":"tc-pd-0","container":"pd","log":"[leadership.go:310] [\"current leadership is deleted\"] [revision=2716] [leader-key=/pd/7270412980872193982/leader] [purpose=\"leader election\"]"}
2023-08-23T16:12:23+08:00 {"namespace":"endless-ha-test-tps-2101058-1-152","level":"INFO","pod":"tc-pd-0","container":"pd","log":"[leadership.go:310] [\"current leadership is deleted\"] [revision=2965] [leader-key=/pd/7270412980872193982/leader] [purpose=\"leader election\"]"}
2023-08-23T16:12:23+08:00 {"level":"INFO","container":"pd","pod":"tc-pd-2","namespace":"endless-ha-test-tps-2101058-1-152","log":"[leadership.go:310] [\"current leadership is deleted\"] [revision=2965] [leader-key=/pd/7270412980872193982/leader] [purpose=\"leader election\"]"}

2023-08-23T16:33:19+08:00 {"container":"pd","level":"INFO","namespace":"endless-ha-test-tps-2101058-1-152","log":"[raft.go:771] [\"3da436621ab53dd7 became leader at term 5\"]","pod":"tc-pd-1"}
2023-08-23T16:33:35+08:00 {"container":"pd","level":"INFO","namespace":"endless-ha-test-tps-2101058-1-152","log":"[leadership.go:310] [\"current leadership is deleted\"] [revision=3926] [leader-key=/pd/7270412980872193982/leader] [purpose=\"leader election\"]","pod":"tc-pd-1"}
2023-08-23T16:33:37+08:00 {"namespace":"endless-ha-test-tps-2101058-1-152","level":"INFO","pod":"tc-pd-0","container":"pd","log":"[leadership.go:310] [\"current leadership is deleted\"] [revision=3930] [leader-key=/pd/7270412980872193982/leader] [purpose=\"leader election\"]"}
2023-08-23T16:38:16+08:00 {"namespace":"endless-ha-test-tps-2101058-1-152","level":"INFO","pod":"tc-pd-0","container":"pd","log":"[leadership.go:310] [\"current leadership is deleted\"] [revision=4121] [leader-key=/pd/7270412980872193982/leader] [purpose=\"leader election\"]"}
2023-08-23T16:38:16+08:00 {"container":"pd","level":"INFO","namespace":"endless-ha-test-tps-2101058-1-152","log":"[leadership.go:310] [\"current leadership is deleted\"] [revision=4121] [leader-key=/pd/7270412980872193982/leader] [purpose=\"leader election\"]","pod":"tc-pd-1"}
```
![image](https://github.com/tikv/pd/assets/19542290/b78d9782-aba7-41c0-8dcd-349104d57413)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.