EpistasisLab / EpistasisLab/tpot

Long evaluation in _pre_test hangs search

Open
#1,023 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Jupyter Notebook
Stars
10.1k
Forks
1.6k
PR merge metrics
No merged PRs in 30d

Description

Thanks for the great tool. When using it, I ran into a bit of
an edge case. For some operators, the
reasonable `_pre_test` decorator (`tpot/decorators.py`)
can actually take an arbitrarily long amount of time, effectively
hanging the search (since it never moves past generating candidates).
The small sampled down `pretest_X` might be reasonable, but
for some operators complexity scales with columns (and it may be
hard to determine the correct number to sample, if any).

You can reproduce the issue by using the following example:
```
> cat example.py

import tpot
import numpy as np
import pmlb

X, y = pmlb.fetch_data("Hill_Valley_without_noise", return_X_y=True)

clf = tpot.TPOTClassifier(
max_time_mins=1,
max_eval_time_mins=1,
cv=2,
config_dict={
'sklearn.decomposition.SparsePCA':{},
'sklearn.linear_model.LogisticRegression':{},
},
verbosity=3,
)

clf.fit(X, y)
```

This will timeout, rather than finish appropriately

```
timeout 5m python example.py # will hang (and get killed 5 minutes after)
echo $? # 124 -> timeout
```

## Possible fix

I have a simple patch that adds a timeout to the
`.fit` call taking place at line 88 in `tpot/decorators.py`. I can PR it if this seems useful.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.