Audit performance of Spark Reads in Hudi 1.X
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
There are a few opportunities for better performance that we have noted while working through other features.
1. We should make `isSplitable` in the FileFormat interface based on whether the path is only a base file. This will allow us to parallelize reads of large files, even in some cases of MoR real-time queries. For MoR read-optimized this should always kick in and give users better parallelization.
2. Returning batches in spark should similarly be controlled by whether there are only base files read. This means MoR read-optimized queries can use this feature.
3. Explore vectorized read support for FileGroupReader. Currently the file group reader path is always manually disabling the vectorized reader so we should explore what it will take to get this supported.
4. Explore columnar batch support for FileGroupReader. Can we convert our iterator of rows into a columnar batch? This would allow us to read more optimally for tables where only a handful of file groups have log files.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-9674
- Type: Improvement
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the Spark read path around the FileFormat interface and FileGroupReader. Review how isSplitable, batch returns, vectorized reads, and columnar batches are currently handled, then identify benchmarks or tests covering MoR read-optimized queries. Done means the viable performance improvements are implemented and their effects are validated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100