Enable parquet enableVectorizedReader for spark incremental read to prevent pef regression
Open
engine:spark
from-jira
priority:critical
status:pr-available
type:improvement
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
now we disable parquet enableVectorizedReader for mor incremental read，
and set "spark.sql.parquet.recordLevelFilter.enabled" = "true" to achieve data filter
which is slow
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-3742
- Type: Improvement
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with Hudi's Spark incremental-read handling and the Parquet settings mentioned in the issue, especially the disabled enableVectorizedReader and enabled spark.sql.parquet.recordLevelFilter.enabled option. Compare the current behavior with the intended incremental-read path; done means vectorized reading is enabled without reproducing the reported performance regression.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100