Using PyTorch models for training and hyperparameter tuning with Dask
- Dominant language
- Python
- Stars
- 951
- Forks
- 262
- PR merge metrics
- No merged PRs in 30d
Description
I was speaking with a group today who is using Dask for I/O and preprocessing of a larger-than-memory dataset and then wants to train a single pytorch model on that large dataset. My initial thought was to look at using Dask-ML's `Incremental` meta-estimator, however after reading the docs more closely I realized that `Incremental` only supports models with a `partial_fit` method.
I'm wondering what are some best practices for training a pytorch model (or running a hyperparameter optimization) on a large Dask collection today? Should users use [skorch](https://skorch.readthedocs.io) to wrap their pytorch models for a scikit-learn compatible API? Is there some other approach users should take?
cc'ing @stsievert as you may have thoughts on this topic
Contributor guide
Assessment
This issue has not been assessed yet.