alteryx / alteryx/evalml

broadcast=True for .scatter() in DaskEngine

Open
#2,095 1 comment 0 reactions 1 assignee Claimed by @asniyaz View on GitHub
enhancement good first issue performance
Dominant language
Python
Stars
850
Forks
96
PR merge metrics
No merged PRs in 30d

Description

We should return to reevaluate the performance implications of broadcasting input data to all workers, all the time in the DaskEngine impl. I would assume Dask would block on completion of the data reaching the workers before it moves on to using the data in its task graphs, so its conceivable that we might be forcing Dask to slow down.

Studying this might be somewhat challenging, as a real world study would probably involve having some Dask cluster that exists where the workers are separated over a network from the scheduler so that there's some latency associated with large amounts of data moving to workers. But ideally, the ideal outcome of this issue involves a test using large dataset, a cluster with multiple workers and executing some not particularly compute expensive computation on the data. We'd like to test the timing difference of broadcast=True vs. broadcast=False.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.