AbsaOSS / AbsaOSS/spark-partition-sizing
In `DataFramePartitioner.DataFrameFunctions.repartitionByRecordCount` use max instead of average to decide about partitoing
Open
enhancement
- Dominant language
- Scala
- Stars
- 9
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
## Background
In `DataFramePartitioner.DataFrameFunctions.repartitionByRecordCount` the limit is compared against the average of records per partition. That can be under the limit while some partitions are still over the limit.
## Feature
Change the method to decide about portioning based on the `max` of record counts per partition if to do or not the repartitioing.
Or make this parametrized or a dedicated method.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.