Optimize train_test_split performance
Open
- Dominant language
- Python
- Stars
- 951
- Forks
- 262
- PR merge metrics
- No merged PRs in 30d
Description
Followup to https://github.com/dask/dask-ml/pull/172
With that PR, `train_test_split` internally uses `ShuffleSplit` to generate the dask array containing the indices to slice. It's possible that the index array isn't on the same machine as the data array.
Each of the A-like branches has the potential for communication.
I think the solution is to generate the indexes and slice in the same task.
Contributor guide
Assessment
This issue has not been assessed yet.