Supporting `Merge` operation to merge the data from each partition that are already ordered.
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Enhancement
After we support dynamic pruning, we deal with the partitions from the partition table through one Reader executor.
The reader doesn't know the information about the partition. So if we want to do things like `... from partitioned_table order by index_col limit 1`, the reader cannot output the data by order of the `index_col`.
If we want to handle the order property, we need to handle the partition's requests simultaneously and add a merge operator to operate as the priority queue to handle the data from each partition. Currently, the reader just sends the request from beginning to end. So it just handles each partition at one time.
We've done a simple optimization in that we read the data from each partition in order, but we don't do the merge. We just do a global Top-N above the reader. We will meet its limitation in real word workloads. So we still need to support the merging operation for the partition table.
Contributor guide
Assessment
This issue has not been assessed yet.