apache / apache/paimon

[Feature] Support rowid pushdown for _ROW_ID filters

Open
#6,744 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Motivation

After row tracking support, users can select the row tracking table this:
`SELECT * FROM t where _ROW_ID = 100;`

Currently it will not take advantage of the table index rowid pushdown facilities, and just process the predicate by FilterOperator (In spark).

If the predicate can be transformed to a `RoaringBitmap32` and passed down to the file format reader, it will hugely improve the scan performance

### Solution

_No response_

### Anything else?

_No response_

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing how Spark's FilterOperator handles _ROW_ID predicates and how the table index and file format reader expose rowid pushdown. The work is done when the predicate can become a RoaringBitmap32 passed to the reader instead of being processed only by FilterOperator, with scan performance verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.