dask / dask/distributed

Running gridsearchcv on dask cluster gives error "Truth of Delayed objects is not supported"

Open
#2,314 8 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

I am running dask on 3 node gcp cluster, using hadoop file storage. On running gridsearchcv (both sklearn and dask_ml) on a 4MB file, it gives error "Truth of Delayed objects is not supported".

Here is my code -

import dask
from dask.distributed import Client
client = Client('scheduler-ip:8786')
import dask.dataframe as dd
df = dd.read_csv('hdfs://path to hdfs')

X = df[indpt_vars_list]
Y = df[target_var]
X, Y= dask.persist(X, Y)

from sklearn.svm import SVC
from sklearn.grid_search import GridSearchCV

svc = SVC(random_state = 0)
param_grid = {
'C' : [0.001, 0.1, 1.0, 2.5, 10.0],
'kernel' : ['rbf', 'poly', 'linear']
}
grid_search = GridSearchCV(svc, param_grid, verbose=2, cv=5, n_jobs=-1)
grid_search.fit(X, Y)

**It gives error : "Truth of Delayed objects is not supported"**

I tried sklearn gridsearchcv and dask in parallel backend, but it is also giving same error
Code :
import dask_ml.joblib
from sklearn.externals import joblib
with joblib.parallel_backend('dask'):
grid_search.fit(X, Y)

Finally on using gridsearchcv provided in dask_searchcv library
from dask_searchcv import GridSearchCV
dask_gridsearchcv = GridSearchCV(svc, param_grid, n_jobs=-1)
dask_gridsearchcv.fit(X, Y)

This doesn't gives any error, but kernel is running for indefinite time.

**Note :** everything works fine if I create a in memory dataset using following code
from sklearn.datasets import make_classification

X, Y = make_classification(n_samples=10000,
n_features=500,
n_classes=2,
n_redundant=250,
random_state=42)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.