AbsaOSS / AbsaOSS/spark-partition-sizing
In `DataFramePartitioner.DataFrameFunctions.repartitionByRecordCount` use max instead of average to decide about partitoing
Ouverte
enhancement
- Langage dominant
- Scala
- Étoiles
- 9
- Forks
- 2
- Métriques de merge des PR
- Aucune PR mergée en 30 j
Description
## Background
In `DataFramePartitioner.DataFrameFunctions.repartitionByRecordCount` the limit is compared against the average of records per partition. That can be under the limit while some partitions are still over the limit.
## Feature
Change the method to decide about portioning based on the `max` of record counts per partition if to do or not the repartitioing.
Or make this parametrized or a dedicated method.
Guide de contribution
Aucun guide de contribution indexé pour ce dépôt
Évaluation
Cette issue n'a pas encore été évaluée.