pingcap / pingcap/tidb

Supporting `Merge` operation to merge the data from each partition that are already ordered.

Open
#40,748 0 comments 0 reactions 0 assignees View on GitHub
sig/execution type/enhancement
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Enhancement

After we support dynamic pruning, we deal with the partitions from the partition table through one Reader executor.

The reader doesn't know the information about the partition. So if we want to do things like `... from partitioned_table order by index_col limit 1`, the reader cannot output the data by order of the `index_col`.

If we want to handle the order property, we need to handle the partition's requests simultaneously and add a merge operator to operate as the priority queue to handle the data from each partition. Currently, the reader just sends the request from beginning to end. So it just handles each partition at one time.

We've done a simple optimization in that we read the data from each partition in order, but we don't do the merge. We just do a global Top-N above the reader. We will meet its limitation in real word workloads. So we still need to support the merging operation for the partition table.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.