EpistasisLab / EpistasisLab/tpot

max_eval_time_mins parameter doesn't stop a long-running eval

Open
#508 16 comments 0 reactions 0 assignees View on GitHub
being worked on bug
Dominant language
Jupyter Notebook
Stars
10.1k
Forks
1.6k
PR merge metrics
No merged PRs in 30d

Description

During `fit()`, I noticed that some evaluations were running for over an hour even though the `max_eval_time_mins=5`.
This is because the code is not actually stopping the thread doing the eval.

In `class Interruptable_cross_val_score`, `stop()` is not actually interrupting the thread, but instead calling `self._stopevent.set()`, which doesn't stop the thread because nothing is checking that event, and then waiting for the thread to stop on its own.

Python doesn't have a good way to interrupt threads. See the discussion at https://stackoverflow.com/questions/323972/is-there-any-way-to-kill-a-thread-in-python

Given that `_wrapped_cross_val_score()` is already running in a separate process due to joblib, one solution would be to make `Interruptable_cross_val_score` a daemon thread, and then remove the call to `tmp_it.stop()` in `_wrapped_cross_val_score()`. Thus when the timeout passes, the process will exit cleanly.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.