EpistasisLab / EpistasisLab/tpot

TPOT freezing at 0% with n_jobs >4 on linux with large dataset

Open
#876 8 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Jupyter Notebook
Stars
10.1k
Forks
1.6k
PR merge metrics
No merged PRs in 30d

Description

Hello,

I am trying to get TPOT running since a while now but always encounter the same errro. I have a linux machine with 24 kernels. When I run TPOT on a large dataset (~6mio rows, ~20 features) it freezes at 0% and after about 10-20 minutes the CPU goes down to a few percent. I already tried setting the multiprocessing to forkserver without any changes. I also tried the dask implementation, but since the max_eval_time_mins does not seem to work there, it runs forever.

However, the problem does not occur when n_jobs != 1 but just if n_jobs > 4. I do not really know what else to try and I would appreciate any suggestions.

Thanks!

```ruby
aml_tpot = TPOTRegressor(scoring = 'neg_mean_squared_error',
generations=20,
population_size=50,
verbosity=3,
random_state = RANDOM_SEED,
n_jobs = 16,
max_eval_time_mins = 20,
cv = 3,
)

aml_tpot.fit(X_train.values, y_train.values.ravel())
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.