joblib.parallel_backend doesn't return when used with dask-mpi and sklearn
- Dominant language
- Python
- Stars
- 951
- Forks
- 262
- PR merge metrics
- No merged PRs in 30d
Description
Although the following seems to run to completion when run with `mpiexec -np 4 script.py`, the context manager doesn't seem to return:
```python3
from dask_mpi import initialize
initialize()
from distributed import Client
client = Client()
import joblib
import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import GridSearchCV
from sklearn.svm import SVC
digits = load_digits()
param_space = {
'C': np.logspace(-3, 3, 5),
'gamma': np.logspace(-4, 4, 3),
}
model = SVC(kernel='rbf')
search = GridSearchCV(model, param_space, cv=3, verbose=10)
with joblib.parallel_backend('dask'):
search.fit(digits.data, digits.target)
print('done') # this is never printed
```
Any thoughts why? I'm using python 3.7.3, scikit-learn 0.21.3, joblib 0.14.0, dask 2.7.0, and dask-mpi 2.0.0 with openmpi 4.0.2 on Ubuntu 18.04.
The motivation behind the above, incidentally, is to examine the feasibility of using dask-mpi to speed up cross validation of models with lengthy training times that conform to the sklearn estimator interface on clusters of machines.
Contributor guide
Assessment
This issue has not been assessed yet.