Optimise or allow for broadcast
- Lenguaje dominante
- Scala
- Estrellas
- 33
- Forks
- 16
- Métricas de merge de PR
- Sin PR fusionados en 30 d
Descripción
"Automatic broadcast on join tables <10MB happens in spark 2.x + already, this covers a lot of the tables already"
References:
- https://stackoverflow.com/questions/43984068/does-spark-sql-autobroadcastjointhreshold-work-for-joins-using-datasets-join-op
- https://jaceklaskowski.gitbooks.io/mastering-spark-sql/spark-sql-joins-broadcast.html
In short:
> "Spark SQL uses broadcast join instead of hash join to optimize join queries when the size of one side data is below spark.sql.autoBroadcastJoinThreshold." "Spark will use autoBroadcastJoinThreshold and automatically broadcast data"
> "Broadcasting large objects is unlikely provide any performance boost, and in practice will often degrade performance and result in stability issue. Remember that broadcasted object has to be first fetch to driver, then send to each worker, and finally loaded into memory."
Guía de contribución
Evaluación
Este issue todavía no se ha evaluado.