apache / apache/fluss

[Client] Support prefix lookup with partial partition key columns

Open
#1,656 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Motivation

Currently, we enforce a strict limitation that prefix lookup columns must encompass all partition fields when querying partitioned tables. However, in numerous practical scenarios, users require the flexibility to perform prefix lookups across multiple partitions. A common use case is executing delta joins where join keys originate from different partitions.

### Solution

- Concurrent Multi-Partition Prefix Lookup: Implement support for concurrent prefix lookups across multiple partitions, with automatic result merging.
- Partition Pruning with Predicate Pushdown in PrefixLookup: Enable partition pruning based on pushed-down predicates, including both JOIN conditions and WHERE conditions, to minimize unnecessary data scanning and improve query performance.
- Currently, to avoid duplicate work related to generic partition filter pushdown based on the predicate system (see https://github.com/apache/fluss/pull/420), this issue will not implement pushdown of WHERE conditions into PrefixKeyLookuper. This functionality will be addressed after https://github.com/apache/fluss/pull/420 is merged.

### Anything else?

_No response_

### Willingness to contribute

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Begin by reading the PrefixKeyLookuper entry point and the predicate pushdown context referenced in PR 420. Completion means supporting concurrent prefix lookups across relevant partitions with merged results and pruning from pushed-down JOIN conditions, while leaving WHERE-condition pushdown for the later work described.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.