pingcap / pingcap/tiflash

Proposal: support union all only by manipulating plan.

Open
#1,829 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

type/feature-request
Dominant language
C++
Stars
1k
Forks
423
Avg merge
1d 15h
Merged PRs (30d)
24

Description

To support "union all", there are two alternatives: support union all operator on TiFlash, or transfer the plans on TiDB.

For fast implemetation, I implement the union all by multiplying the plans. For expamle, given a sql:

(select * from x1) union all (select * from x2) union all (select * from x3)

It's equal to three (maybe more) flattened MPP tasks:

  1. select * from x1
  2. select * from x2
  3. select * from x3

We take the advantage that the final "MppGather" has taken the responsibilities of doing "union all". For a more complicated example:

select * from (select * from x1 union all select * from x2) X join (select * from x3 union all select * from x4) Y on X.a = Y.a

For this case, what first comes to mind is to do union all crossly, say union all (x1 join x3) (x1 join x4) (x2 join x3) (x2 join x4). That's a disaster if the branches of union increase. There is a way around it. If the subplan of join is union all, always do exchange before union all. For this statement, if we adopt shuffled hash join, the final tasks are:

1. (Exchange Recv(task 2, 3)) join (Exchange Recv(task 4, 5))
2. select * from x1 then Exchange Sender shuffling and sending data to task1 
3. select * from x2 then Exchange Sender shuffling and sending data to task1
4. select * from x3 then Exchange Sender shuffling and sending data to task1 
5. select * from x4 then Exchange Sender shuffling and sending data to task1

If we choose broadcast join, the tasks will be

1. x1 join (Exchange Recv(task 3, 4))
2. x2 join (Exchange Recv(task 3, 4))
3. select * from x3 then Exchange Sender broadcast data to task1, 2 
5. select * from x4 then Exchange Sender broadcast data to task1, 2

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names no files or tests. Start by tracing TiFlash's MppGather and exchange handling for UNION ALL cases; done would require an agreed implementation and validation for the simple, shuffled-hash-join, and broadcast-join examples.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, sql
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.