New hfile reader should support "stream" mode
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
The HFile reader in HBase has two primary modes: # *Pread mode* - This is the first mode that starts with "p"
# *Streaming mode* - This is the second mode that prefixes with "stream"
{*}Pread mode{*}: (point look up, already supported) * Uses random access reads (positioned reads)
* Good for seeking to specific positions in the file
* Optimized for random access patterns
* Typically used for point lookups or when reading non-contiguous parts of a file
* More efficient when you need to jump around in the file
{*}Streaming mode{*}: * Uses sequential reads
* Optimized for reading large contiguous sections of data
* More efficient when reading entire blocks or scanning through data sequentially
* Better performance for scan operations or when reading a file from beginning to end
* Reduces the number of I/O operations
The streaming mode we should add similar support in hudi hfile reader.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-9353
- Type: Improvement
---
## Comments
05/May/25 16:59;daviszhang;next step:
# Need to showcase when batch look up on 1 index file group will happen
# in that case, when we use the old hfile reader streaming mode it out performs the point look up mode;;;
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating Hudi's HFile reader and comparing its existing point-lookup (pread) path with the HBase reader's streaming mode. Use the batch lookup on one index file group described in the comment to measure the behavior; done means streaming mode is supported and its performance benefit is demonstrated for that case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 45/100