apache / apache/hudi

New hfile reader should support "stream" mode

Open
#16,986 1 comment 0 reactions 0 assignees View on GitHub
from-jira priority:high type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

The HFile reader in HBase has two primary modes: # *Pread mode* - This is the first mode that starts with "p"
# *Streaming mode* - This is the second mode that prefixes with "stream"

 
{*}Pread mode{*}: (point look up, already supported) * Uses random access reads (positioned reads)
* Good for seeking to specific positions in the file
* Optimized for random access patterns
* Typically used for point lookups or when reading non-contiguous parts of a file
* More efficient when you need to jump around in the file

{*}Streaming mode{*}: * Uses sequential reads
* Optimized for reading large contiguous sections of data
* More efficient when reading entire blocks or scanning through data sequentially
* Better performance for scan operations or when reading a file from beginning to end
* Reduces the number of I/O operations

 
The streaming mode we should add similar support in hudi hfile reader.

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-9353
- Type: Improvement

---

## Comments

05/May/25 16:59;daviszhang;next step:
# Need to showcase when batch look up on 1 index file group will happen
# in that case, when we use the old hfile reader streaming mode it out performs the point look up mode;;;

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating Hudi's HFile reader and comparing its existing point-lookup (pread) path with the HBase reader's streaming mode. Use the batch lookup on one index file group described in the comment to measure the behavior; done means streaming mode is supported and its performance benefit is demonstrated for that case.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.