EpistasisLab / EpistasisLab/tpot
TPOT freezing at 0% with n_jobs >4 on linux with large dataset
- Dominant language
- Jupyter Notebook
- Stars
- 10.1k
- Forks
- 1.6k
- PR merge metrics
- No merged PRs in 30d
Description
Hello,
I am trying to get TPOT running since a while now but always encounter the same errro. I have a linux machine with 24 kernels. When I run TPOT on a large dataset (~6mio rows, ~20 features) it freezes at 0% and after about 10-20 minutes the CPU goes down to a few percent. I already tried setting the multiprocessing to forkserver without any changes. I also tried the dask implementation, but since the max_eval_time_mins does not seem to work there, it runs forever.
However, the problem does not occur when n_jobs != 1 but just if n_jobs > 4. I do not really know what else to try and I would appreciate any suggestions.
Thanks!
```ruby
aml_tpot = TPOTRegressor(scoring = 'neg_mean_squared_error',
generations=20,
population_size=50,
verbosity=3,
random_state = RANDOM_SEED,
n_jobs = 16,
max_eval_time_mins = 20,
cv = 3,
)
aml_tpot.fit(X_train.values, y_train.values.ravel())
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.