pingcap / pingcap/tidb

Introduce coordinator for mpp tasks execution

Open
#44,032 0 comments 0 reactions 0 assignees View on GitHub
type/enhancement
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Enhancement
The main advatages are:

- Better error tracking mechanism: mpp task can report first error directly to coordinator
- Provide infrastucture to support global runtime filter, coordinator can play the role as merging all hash build infos and broadcast to probe mpp tasks.

## TaskLists
- [x] Refact MPPClient and MPPGather to introduce MPPCoordinator interface
https://github.com/pingcap/tidb/pull/44034
- [x] Modify protocal to prepare for better error tracking
https://github.com/pingcap/tipb/pull/310
https://github.com/pingcap/kvproto/pull/1133
https://github.com/tikv/client-c/pull/148
- [x] Implement better error tracking both in tidb/tiflash side
https://github.com/pingcap/tidb/pull/44697
https://github.com/pingcap/tiflash/pull/7658
- [x] Consider add protection mechanism to support degrade to previous way
- [x] Add new metrics to track mpp coordinator counter and wait report related metrics, and used them as error indicator in some existing endless cases

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.