dask / dask/dask-examples

Create an Example of Using TPOT Using Dataset that DOESN'T Fit in Memory

Open
#227 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
387
Forks
225
PR merge metrics
No merged PRs in 30d

Description

All of the examples I've seen either:

1. Show TPOT using Dask for training on a dataset that fits in memory ([shown here](https://examples.dask.org/machine-learning/tpot.html))
2. Show how to use Dask-ml with `Incremental` to train on a dataset that doesn't fit in memory ([shown here](https://examples.dask.org/machine-learning/training-on-large-datasets.html))

...but not how to use TPOT AND a larger than memory dataset.

My attempt at this looked like this:

```
tpot = TPOTRegressor(generations=100, population_size=25, use_dask=True)

from dask_ml.wrappers import Incremental
inc = Incremental(tpot, scoring='neg_mean_absolute_error')

inc.fit(X_train, y_train)

print(inc.score(X_test.values, y_test.values))
```

...but this of course throws the error:

```
Traceback (most recent call last):
File "Z:\Python_Projects\test5.py", line 94, in
inc.fit(X_train, y_train)
File "C:\Users\chalu\AppData\Roaming\Python\Python310\site-packages\dask_ml\wrappers.py", line 579, in fit
self._fit_for_estimator(estimator, X, y, **fit_kwargs)
File "C:\Users\chalu\AppData\Roaming\Python\Python310\site-packages\dask_ml\wrappers.py", line 561, in _fit_for_estimator
result = estimator.partial_fit(X=X, y=y, **fit_kwargs)
AttributeError: 'TPOTRegressor' object has no attribute 'partial_fit'
```

...because the TPOT objects don't have an incremental fit function.

I've opened an issue [here](https://github.com/EpistasisLab/tpot/issues/1264) re: attempting to train a TPOT regressor on a "larger than memory" dataset using Dask as I don't know if TPOT allows for larger than memory datasets, but this would be an awesome feature to have some day soon.

Thanks!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.