apache / apache/hudi

Audit performance of Spark Reads in Hudi 1.X

Open
#17,128 1 comment 0 reactions 0 assignees View on GitHub
from-jira priority:high type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

There are a few opportunities for better performance that we have noted while working through other features.

1. We should make `isSplitable` in the FileFormat interface based on whether the path is only a base file. This will allow us to parallelize reads of large files, even in some cases of MoR real-time queries. For MoR read-optimized this should always kick in and give users better parallelization.

2. Returning batches in spark should similarly be controlled by whether there are only base files read. This means MoR read-optimized queries can use this feature.

3. Explore vectorized read support for FileGroupReader. Currently the file group reader path is always manually disabling the vectorized reader so we should explore what it will take to get this supported.

4. Explore columnar batch support for FileGroupReader. Can we convert our iterator of rows into a columnar batch? This would allow us to read more optimally for tables where only a handful of file groups have log files.

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-9674
- Type: Improvement

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the Spark read path around the FileFormat interface and FileGroupReader. Review how isSplitable, batch returns, vectorized reads, and columnar batches are currently handled, then identify benchmarks or tests covering MoR read-optimized queries. Done means the viable performance improvements are implemented and their effects are validated.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.