Change partitioner abstraction to implement multiple strategies
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Existing UpsertPartitioner only considers file sizing to assign inserts/updates. We also want to consider data locality and other factors. So change partitioner abstraction to make it easy to implement and plug in other strategies.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-1629
- Type: New Feature
- Epic: https://issues.apache.org/jira/browse/HUDI-1628
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue names the UpsertPartitioner abstraction but no source file or test. Start by locating UpsertPartitioner and read how file sizing currently assigns inserts and updates; done means the abstraction can support and plug in multiple strategies, including data locality.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100