Establish better index lookup and write perf w/ RLI compared to global bloom
Open
area:writer
from-jira
priority:critical
type:improvement
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
With the RLI we are looking to add to MDT, we need to establish better performance compared to global bloom.
Prepare a table w/ 10k or more files and measure the performance
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-5667
- Type: Improvement
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the MDT/RLI and global bloom index entry points or existing benchmark tests in the Hudi repository, then review how performance measurements are recorded. Done means producing a table using at least 10k files that compares RLI and global bloom lookup and write performance.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 45/100