alteryx / alteryx/evalml

broadcast=True for .scatter() in DaskEngine

Abierto
#2,095 1 comentario 0 reacciones 1 asignado Reclamado por @asniyaz Ver en GitHub
enhancement good first issue performance
Lenguaje dominante
Python
Estrellas
850
Forks
96
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

We should return to reevaluate the performance implications of broadcasting input data to all workers, all the time in the DaskEngine impl. I would assume Dask would block on completion of the data reaching the workers before it moves on to using the data in its task graphs, so its conceivable that we might be forcing Dask to slow down.

Studying this might be somewhat challenging, as a real world study would probably involve having some Dask cluster that exists where the workers are separated over a network from the scheduler so that there's some latency associated with large amounts of data moving to workers. But ideally, the ideal outcome of this issue involves a test using large dataset, a cluster with multiple workers and executing some not particularly compute expensive computation on the data. We'd like to test the timing difference of broadcast=True vs. broadcast=False.

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.