Optimise the computation for multiple secondary indexes in HoodieAppendHandle
Open
area:metadata-table
from-jira
priority:high
type:bug
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Currently in HoodieAppendHandle, we are reading the file slice multiple times for multiple secondary indexes. We can instead read the file slice once for all the secondary indexes and then update the stats.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-9533
- Type: Bug
- Fix version(s):
- 1.2.0
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in HoodieAppendHandle and trace how the file slice is read for each secondary index. Confirm the existing statistics-update flow, then make the computation reuse one file-slice read for all secondary indexes and verify that the resulting statistics remain correct.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 65/100