[Feature] Discussion: Can we use Bitmap File Index to update DV when delete where by non-primary column?
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Motivation
https://cwiki.apache.org/confluence/display/PAIMON/PIP-23+Introduce+bitmap+file+index
BitmapIndex tracks row positions of a specific value.
Let's say we have a primarykey table, a non-primary-key column: `type`. And we enable bitmap file index on the `type` column. For Spark, when `DELETE FROM table WHERE type IN 'type1'`, it seems that it'll first scan parquet files to get `FilePath` and `Position of rows`, then shuffle by `FilePath` to update DV. I'm curious if we can directly use the bitmap index to update DV, skipping the file scan and shuffling.
### Solution
_No response_
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading PIP-23, then examine the described Spark DELETE path: scanning parquet files for file paths and row positions, shuffling by file path, and updating deletion vectors. Done means establishing whether the bitmap file index can safely replace that scan and shuffle, and documenting or implementing the resulting design.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100