tikv / tikv/pd

Operator might be ended with the wrong status

Open
#7,839 0 comments 0 reactions 0 assignees View on GitHub
type/enhancement
Dominant language
Go
Stars
1.2k
Forks
783
Avg merge
5d 21h
Merged PRs (30d)
36

Description

## Enhancement Task
https://github.com/tikv/pd/blob/d6d9feab3e2a5180acbcc7095723d43e97798686/pkg/schedule/operator/operator_controller.go#L107-L235

If line 232 is executed first, the operator may be changed to timeout status since it indeed exceeds the max execution time if there are pending heartbeats. When calling line 116, it will be skipped because of the timeout status. But actually, it could be executed successfully on TiKV side.

Here is an example:
```
Feb 21, 2024 @ 12:14:17.011 [operator_controller.go:443] ["add operator"] [region-id=1199980] [operator="\"fix-peer-role {promote peer: store [2803711]} (kind:unknown, region:1199980(5218, 3189), createAt:2024-02-21 12:14:17.00619947 +0800 CST m=+70594.374713221, startAt:0001-01-01 00:00:00 +0000 UTC, currentStep:0, size:56, steps:[0:{promote learner peer 430111713 on store 2803711 to voter}],timeout:[1m0s])\""] [additional-info=]
Feb 21, 2024 @ 12:15:19.462 [region.go:645] ["region ConfVer changed"] [region-id=1199980] [detail="Remove peer:{id:430111713 store_id:2803711 role:Learner },Add peer:{id:430111713 store_id:2803711 }"] [old-confver=3189] [new-confver=3190]
Feb 21, 2024 @ 12:15:19.468 [operator_controller.go:580] ["operator timeout"] [region-id=1199980] [takes=1m2.45689524s] [operator="\"fix-peer-role {promote peer: store [2803711]} (kind:unknown, region:1199980(5218, 3189), createAt:2024-02-21 12:14:17.00619947 +0800 CST m=+70594.374713221, startAt:2024-02-21 12:14:17.01159741 +0800 CST m=+70594.380111161, currentStep:0, size:56, steps:[0:{promote learner peer 430111713 on store 2803711 to voter}],timeout:[1m0s]) timeout\""] [additional-info=]
```

Contributor guide

Open the contributing guide

Research direction

Read pkg/schedule/operator/operator_controller.go around lines 107–235, focusing on the status checks at lines 116 and 232 and the timeout handling shown in the example logs. Trace how pending heartbeats and a successful TiKV-side operation are ordered, then verify that an operator completed successfully is not ended with timeout status.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.