Running gridsearchcv on dask cluster gives error "Truth of Delayed objects is not supported"
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
I am running dask on 3 node gcp cluster, using hadoop file storage. On running gridsearchcv (both sklearn and dask_ml) on a 4MB file, it gives error "Truth of Delayed objects is not supported".
Here is my code -
import dask
from dask.distributed import Client
client = Client('scheduler-ip:8786')
import dask.dataframe as dd
df = dd.read_csv('hdfs://path to hdfs')
X = df[indpt_vars_list]
Y = df[target_var]
X, Y= dask.persist(X, Y)
from sklearn.svm import SVC
from sklearn.grid_search import GridSearchCV
svc = SVC(random_state = 0)
param_grid = {
'C' : [0.001, 0.1, 1.0, 2.5, 10.0],
'kernel' : ['rbf', 'poly', 'linear']
}
grid_search = GridSearchCV(svc, param_grid, verbose=2, cv=5, n_jobs=-1)
grid_search.fit(X, Y)
**It gives error : "Truth of Delayed objects is not supported"**
I tried sklearn gridsearchcv and dask in parallel backend, but it is also giving same error
Code :
import dask_ml.joblib
from sklearn.externals import joblib
with joblib.parallel_backend('dask'):
grid_search.fit(X, Y)
Finally on using gridsearchcv provided in dask_searchcv library
from dask_searchcv import GridSearchCV
dask_gridsearchcv = GridSearchCV(svc, param_grid, n_jobs=-1)
dask_gridsearchcv.fit(X, Y)
This doesn't gives any error, but kernel is running for indefinite time.
**Note :** everything works fine if I create a in memory dataset using following code
from sklearn.datasets import make_classification
X, Y = make_classification(n_samples=10000,
n_features=500,
n_classes=2,
n_redundant=250,
random_state=42)
Contributor guide
Assessment
This issue has not been assessed yet.