Speed up reading of files from S3 in S3EventsIncrSource
Open
area:ingest
from-jira
priority:high
type:improvement
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
In S3EventsIncrSource, we load dataframe of N files using dataframeReader.load(files[]). we can improve the speed of reading S3 files by leveraging spark.parallelize().
Ref issue: https://github.com/apache/hudi/issues/5952
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-4835
- Type: Improvement
Contributor guide
No contributing guide indexed for this repository
Research direction
Locate S3EventsIncrSource and inspect the dataframeReader.load(files[]) call, since the issue does not name a source file or test. Determine how Spark parallelization would affect the file reads, then verify that the source reads the same files with improved performance.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java, spark
- Domain
- data-engineering, stream-processing
- Issue type
- Refactor
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100