SparkSingleFileSortExecutionStrategy can not work for big file
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
SparkSingleFileSortExecutionStrategy extend `SparkSizeBasedClusteringPlanStrategy` will filter by config `hoodie.clustering.plan.strategy.small.file.limit`, it can not work for big file to sort inline row group
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-5149
- Type: Improvement
Contributor guide
No contributing guide indexed for this repository
Research direction
Locate SparkSingleFileSortExecutionStrategy and its SparkSizeBasedClusteringPlanStrategy extension, then inspect how hoodie.clustering.plan.strategy.small.file.limit filters files. Verify the intended behavior for large files that need inline row-group sorting; done means the strategy can handle those files instead of excluding them because they exceed the small-file limit.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100