EpistasisLab / EpistasisLab/tpot

use_dask and joblib don't handle Exceptions

Open
#779 8 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
10.1k
Forks
1.6k
PR merge metrics
No merged PRs in 30d

Description

My issue is described at https://github.com/dask/dask-jobqueue/issues/169

A huge thank to @guillaumeeb who identified the problem!

## Context of the issue

Basically, an error is triggered by the dask backend if a computation encounters exceptions,
and after some attempts to relaunch the task, it makes the `fit` function fail.

## Process to reproduce the issue

Set `use_dask=True` with a dataset that will crash with some pipelines (for example PolynomialFeatures with 800 columns).

## Expected result

Without dask, `_wrapped_cross_val_score` catches the exceptions and returns a -inf score.
The relevant code is there: https://github.com/EpistasisLab/tpot/blob/507b45db01e8f88651f4ce8e03b607a5b50146f5/tpot/gp_deap.py#L434-L480

## Current result

The current result is strange (KeyError), but the error message on the master branch of Dask will indicate that a worker restarted (I think).

## Possible fix

There should be a way to catch the memory error (or worker restart) to return a -inf score.

When I try with a LSFCluster backend, it causes an exception

When I try with a LocalCluster (`Client()`), the whole notebook crashes after some attempts.

## Immediate fix

~~I think using the joblib backend provided by Dask should be fine? I'll try tomorrow!~~

It doesn't work, see below.

## Screenshots

from the dask web interface
![43351582_285330748746423_8179954919642497024_n](https://user-images.githubusercontent.com/13174805/46576664-d42e0880-c9cf-11e8-8e33-e3a4caf23cc3.png)

![43319853_1882248011842149_2283641915238776832_n](https://user-images.githubusercontent.com/13174805/46576665-dbedad00-c9cf-11e8-9385-64a693d4ce70.png)

## Logs

with one worker

traceback: https://pastebin.com/raw/QvtVXkBD
dask-worker.err: https://ptpb.pw/SwbX

Notice that the dask worker restarted 3 times, which appears to be a constant.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.