EpistasisLab / EpistasisLab/tpot
Long evaluation in _pre_test hangs search
- Dominant language
- Jupyter Notebook
- Stars
- 10.1k
- Forks
- 1.6k
- PR merge metrics
- No merged PRs in 30d
Description
Thanks for the great tool. When using it, I ran into a bit of
an edge case. For some operators, the
reasonable `_pre_test` decorator (`tpot/decorators.py`)
can actually take an arbitrarily long amount of time, effectively
hanging the search (since it never moves past generating candidates).
The small sampled down `pretest_X` might be reasonable, but
for some operators complexity scales with columns (and it may be
hard to determine the correct number to sample, if any).
You can reproduce the issue by using the following example:
```
> cat example.py
import tpot
import numpy as np
import pmlb
X, y = pmlb.fetch_data("Hill_Valley_without_noise", return_X_y=True)
clf = tpot.TPOTClassifier(
max_time_mins=1,
max_eval_time_mins=1,
cv=2,
config_dict={
'sklearn.decomposition.SparsePCA':{},
'sklearn.linear_model.LogisticRegression':{},
},
verbosity=3,
)
clf.fit(X, y)
```
This will timeout, rather than finish appropriately
```
timeout 5m python example.py # will hang (and get killed 5 minutes after)
echo $? # 124 -> timeout
```
## Possible fix
I have a simple patch that adds a timeout to the
`.fit` call taking place at line 88 in `tpot/decorators.py`. I can PR it if this seems useful.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.