EpistasisLab / EpistasisLab/tpot
use_dask and joblib don't handle Exceptions
- Dominant language
- Jupyter Notebook
- Stars
- 10.1k
- Forks
- 1.6k
- PR merge metrics
- No merged PRs in 30d
Description
My issue is described at https://github.com/dask/dask-jobqueue/issues/169
A huge thank to @guillaumeeb who identified the problem!
## Context of the issue
Basically, an error is triggered by the dask backend if a computation encounters exceptions,
and after some attempts to relaunch the task, it makes the `fit` function fail.
## Process to reproduce the issue
Set `use_dask=True` with a dataset that will crash with some pipelines (for example PolynomialFeatures with 800 columns).
## Expected result
Without dask, `_wrapped_cross_val_score` catches the exceptions and returns a -inf score.
The relevant code is there: https://github.com/EpistasisLab/tpot/blob/507b45db01e8f88651f4ce8e03b607a5b50146f5/tpot/gp_deap.py#L434-L480
## Current result
The current result is strange (KeyError), but the error message on the master branch of Dask will indicate that a worker restarted (I think).
## Possible fix
There should be a way to catch the memory error (or worker restart) to return a -inf score.
When I try with a LSFCluster backend, it causes an exception
When I try with a LocalCluster (`Client()`), the whole notebook crashes after some attempts.
## Immediate fix
~~I think using the joblib backend provided by Dask should be fine? I'll try tomorrow!~~
It doesn't work, see below.
## Screenshots
from the dask web interface


## Logs
with one worker
traceback: https://pastebin.com/raw/QvtVXkBD
dask-worker.err: https://ptpb.pw/SwbX
Notice that the dask worker restarted 3 times, which appears to be a constant.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.