dask / dask/dask-ml

Using PyTorch models for training and hyperparameter tuning with Dask

Open
#836 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
951
Forks
262
PR merge metrics
No merged PRs in 30d

Description

I was speaking with a group today who is using Dask for I/O and preprocessing of a larger-than-memory dataset and then wants to train a single pytorch model on that large dataset. My initial thought was to look at using Dask-ML's `Incremental` meta-estimator, however after reading the docs more closely I realized that `Incremental` only supports models with a `partial_fit` method.

I'm wondering what are some best practices for training a pytorch model (or running a hyperparameter optimization) on a large Dask collection today? Should users use [skorch](https://skorch.readthedocs.io) to wrap their pytorch models for a scikit-learn compatible API? Is there some other approach users should take?

cc'ing @stsievert as you may have thoughts on this topic

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.