Refine MPP Task abort logic: Let mpp task also send a cancel signal to downstream when it fails
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 1k
- Forks
- 423
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 24
Description
Enhancement
current abort logic
Currently, when an mpp task fails, it only sends an error signal to its upstream, i.e. send from its ExchangeSender to upstream's ExchangeReceiver, and the upstream can only get this signal by calling read.
And only after tidb-server(task -1) get this error signal, it sends CancelMPPTask rpc requests to every worker node. Then TiFlash will abort the queries.
problem and example
For example, consider this query:
task -1 (tidb)
|
|
task 1 (have a join)
|(probe) \ (build)
| \
| \
task 2 task 3
|
|
|
task 4
- task 2 failed, it want to push an error to task 1.
- note that task 4 can't receive this signal, so it will execute until buffer full.
- task 1 has a CreatingSetsBlockInputStream(join executor), it will call build-side's
readfirst until task 3 end, then call probe-side'sread. - so task 3 and task 4 (both should ended sooner) have to run longer. This leads to a waste of resources.
Refine suggestions
- send error signal to it's downstream, in this example, task 2 should notify task 4.
- directly notify coordinator(can be tidb or a selected tiflash node), then coordinator notify every work node.
- a related issue: https://github.com/pingcap/tiflash/issues/6390
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the MPP task abort path through ExchangeSender, ExchangeReceiver, CreatingSetsBlockInputStream, and the CancelMPPTask RPC described in the issue. Determine whether downstream tasks or a coordinator should receive failure cancellation, then define completion as preventing dependent tasks from running after an upstream failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100