apache / apache/paimon

[Feature] Compare batch maxKey to reduce comparisons in SortMergeReader

Open
#753 1 comment 0 reactions 1 assignee Claimed by @schnappi17 View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.

### Motivation

In SortMergeReader, each sub reader is batched reader.

When adding a new batch to the priority queue, we can look at the maximum key of the batch, and if its maximum key is smaller than the minimum key of other batches, then we can just output the whole batch.

### Solution

_No response_

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.