AbsaOSS / AbsaOSS/spark-partition-sizing
In `DataFramePartitioner.DataFrameFunctions.repartitionByRecordCount` use max instead of average to decide about partitoing
Abierto
enhancement
- Lenguaje dominante
- Scala
- Estrellas
- 9
- Forks
- 2
- Métricas de merge de PR
- Sin PR fusionados en 30 d
Descripción
## Background
In `DataFramePartitioner.DataFrameFunctions.repartitionByRecordCount` the limit is compared against the average of records per partition. That can be under the limit while some partitions are still over the limit.
## Feature
Change the method to decide about portioning based on the `max` of record counts per partition if to do or not the repartitioing.
Or make this parametrized or a dedicated method.
Guía de contribución
No hay ninguna guía de contribución indexada para este repositorio
Evaluación
Este issue todavía no se ha evaluado.