[Client] Support prefix lookup with partial partition key columns
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Motivation
Currently, we enforce a strict limitation that prefix lookup columns must encompass all partition fields when querying partitioned tables. However, in numerous practical scenarios, users require the flexibility to perform prefix lookups across multiple partitions. A common use case is executing delta joins where join keys originate from different partitions.
### Solution
- Concurrent Multi-Partition Prefix Lookup: Implement support for concurrent prefix lookups across multiple partitions, with automatic result merging.
- Partition Pruning with Predicate Pushdown in PrefixLookup: Enable partition pruning based on pushed-down predicates, including both JOIN conditions and WHERE conditions, to minimize unnecessary data scanning and improve query performance.
- Currently, to avoid duplicate work related to generic partition filter pushdown based on the predicate system (see https://github.com/apache/fluss/pull/420), this issue will not implement pushdown of WHERE conditions into PrefixKeyLookuper. This functionality will be addressed after https://github.com/apache/fluss/pull/420 is merged.
### Anything else?
_No response_
### Willingness to contribute
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Begin by reading the PrefixKeyLookuper entry point and the predicate pushdown context referenced in PR 420. Completion means supporting concurrent prefix lookups across relevant partitions with merged results and pruning from pushed-down JOIN conditions, while leaving WHERE-condition pushdown for the later work described.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100