Fix deletion of entire record in MDT for col stats, bloom filter
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
w/ RLI, we are introducing a proper way to delete a MDT record. [https://github.com/apache/hudi/pull/9058]
We might have to follow similar logic for other partitions as well to optimize it better. We should avoid relying on nested fields to deduce whether a record is deleted (for eg,
ColumnStatsMetadata.isDeleted
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-6461
- Type: Improvement
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the linked RLI pull request and tracing the MDT deletion paths for column statistics and bloom-filter partitions. Inspect how ColumnStatsMetadata.isDeleted is used; done means relevant partitions use proper entire-record deletion rather than inferring deletion from nested fields.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 38/100