alteryx / alteryx/evalml

broadcast=True for .scatter() in DaskEngine

Offen
#2,095 1 Kommentar 0 Reaktionen 1 zugewiesene Person Beansprucht von @asniyaz Auf GitHub ansehen
enhancement good first issue performance
Vorherrschende Sprache
Python
Sterne
850
Forks
96
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

We should return to reevaluate the performance implications of broadcasting input data to all workers, all the time in the DaskEngine impl. I would assume Dask would block on completion of the data reaching the workers before it moves on to using the data in its task graphs, so its conceivable that we might be forcing Dask to slow down.

Studying this might be somewhat challenging, as a real world study would probably involve having some Dask cluster that exists where the workers are separated over a network from the scheduler so that there's some latency associated with large amounts of data moving to workers. But ideally, the ideal outcome of this issue involves a test using large dataset, a cluster with multiple workers and executing some not particularly compute expensive computation on the data. We'd like to test the timing difference of broadcast=True vs. broadcast=False.

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.