pingcap / pingcap/tiflash

Refine MPP Task abort logic: Let mpp task also send a cancel signal to downstream when it fails

Open
#6,389 0 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

component/compute help wanted type/enhancement
Dominant language
C++
Stars
1k
Forks
423
Avg merge
1d 15h
Merged PRs (30d)
24

Description

Enhancement

current abort logic

Currently, when an mpp task fails, it only sends an error signal to its upstream, i.e. send from its ExchangeSender to upstream's ExchangeReceiver, and the upstream can only get this signal by calling read.

And only after tidb-server(task -1) get this error signal, it sends CancelMPPTask rpc requests to every worker node. Then TiFlash will abort the queries.

problem and example

For example, consider this query:

task -1 (tidb)
|
|
task 1 (have a join)
|(probe)        \ (build)
|                \
|                 \
task 2             task 3
|
|
|
task 4
  1. task 2 failed, it want to push an error to task 1.
  2. note that task 4 can't receive this signal, so it will execute until buffer full.
  3. task 1 has a CreatingSetsBlockInputStream(join executor), it will call build-side's read first until task 3 end, then call probe-side's read.
  4. so task 3 and task 4 (both should ended sooner) have to run longer. This leads to a waste of resources.
Refine suggestions
  • send error signal to it's downstream, in this example, task 2 should notify task 4.
  • directly notify coordinator(can be tidb or a selected tiflash node), then coordinator notify every work node.
  • a related issue: https://github.com/pingcap/tiflash/issues/6390

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the MPP task abort path through ExchangeSender, ExchangeReceiver, CreatingSetsBlockInputStream, and the CancelMPPTask RPC described in the issue. Determine whether downstream tasks or a coordinator should receive failure cancellation, then define completion as preventing dependent tasks from running after an upstream failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.