RLI metric does not work for RDD due to lazy evaluation
Open
from-jira
priority:high
type:bug
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
org.apache.hudi.metadata.HoodieBackedTableMetadata#readRecordIndexWithMapping
if underneath it is rdd based API calls, the metric does not work due to rdd is lazily evaluated.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-9544
- Type: Bug
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at org.apache.hudi.metadata.HoodieBackedTableMetadata#readRecordIndexWithMapping and trace the RDD-based API calls described in the issue. Read the linked HUDI-9544 context to determine how the RLI metric should behave, then verify that it is recorded correctly despite RDD lazy evaluation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100