Introduce coordinator for mpp tasks execution
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Enhancement
The main advatages are:
- Better error tracking mechanism: mpp task can report first error directly to coordinator
- Provide infrastucture to support global runtime filter, coordinator can play the role as merging all hash build infos and broadcast to probe mpp tasks.
## TaskLists
- [x] Refact MPPClient and MPPGather to introduce MPPCoordinator interface
https://github.com/pingcap/tidb/pull/44034
- [x] Modify protocal to prepare for better error tracking
https://github.com/pingcap/tipb/pull/310
https://github.com/pingcap/kvproto/pull/1133
https://github.com/tikv/client-c/pull/148
- [x] Implement better error tracking both in tidb/tiflash side
https://github.com/pingcap/tidb/pull/44697
https://github.com/pingcap/tiflash/pull/7658
- [x] Consider add protection mechanism to support degrade to previous way
- [x] Add new metrics to track mpp coordinator counter and wait report related metrics, and used them as error indicator in some existing endless cases
Contributor guide
Assessment
This issue has not been assessed yet.