apache / apache/paimon

[Feature] Discussion: Can we use Bitmap File Index to update DV when delete where by non-primary column?

Open
#7,189 2 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Motivation

https://cwiki.apache.org/confluence/display/PAIMON/PIP-23+Introduce+bitmap+file+index
BitmapIndex tracks row positions of a specific value.
Let's say we have a primarykey table, a non-primary-key column: `type`. And we enable bitmap file index on the `type` column. For Spark, when `DELETE FROM table WHERE type IN 'type1'`, it seems that it'll first scan parquet files to get `FilePath` and `Position of rows`, then shuffle by `FilePath` to update DV. I'm curious if we can directly use the bitmap index to update DV, skipping the file scan and shuffling.

### Solution

_No response_

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading PIP-23, then examine the described Spark DELETE path: scanning parquet files for file paths and row positions, shuffling by file path, and updating deletion vectors. Done means establishing whether the bitmap file index can safely replace that scan and shuffle, and documenting or implementing the resulting design.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.