[RFC-87] Support collecting column range metadata for AVRO log block efficiently
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Currently, column range metadata for column stats index are collected by an individual iterating and extracting operation, which is inefficient. An more efficient way is needed to make it solid for enable by default for Flink writer.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-9191
- Type: Sub-task
- Parent: https://issues.apache.org/jira/browse/HUDI-9075
- Fix version(s):
- 1.2.0
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the column stats index and AVRO log block context referenced in the issue, then trace how the Flink writer currently collects column range metadata through individual iteration and extraction. Done means an efficient collection approach is defined and enabled by default for the Flink writer, with behavior aligned to JIRA HUDI-9191.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100