dask / dask/distributed

Use dask_ml GridSearchCV on HPC cluster client

Open
#3,463 14 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Hi,
I'm working on a large data-set and I try to find my model hyperparameters thanks to GridSearchCV from dask_ml as presented in the [dask_ml tutorial](https://ml.dask.org/joblib.html)

here is my python code :
```python
import pandas as pd
from sklearn.model_selection import GroupKFold
from dask_ml.model_selection import GridSearchCV
from dask.distributed import Client, progress
from dask_jobqueue import PBSCluster
from sklearn.externals import joblib

dataset_path = "/data_test.sqlite"
data_field = "code"
layer_name = "data_test"
cv_folds = 5
cv_parameters = {'n_estimators': [50, 100, 150]}

features_labels = [
'sentinel2_ndvi_20180101', 'sentinel2_ndvi_20180111',
'sentinel2_ndvi_20180121', 'sentinel2_ndvi_20180131',
'sentinel2_ndvi_20180210', 'sentinel2_ndvi_20180220',
'sentinel2_ndvi_20180302', 'sentinel2_ndvi_20180312',
'sentinel2_ndvi_20180322', 'sentinel2_ndvi_20180401',
'sentinel2_ndvi_20180411', 'sentinel2_ndvi_20180421',
'sentinel2_ndvi_20180501', 'sentinel2_ndvi_20180511',
'sentinel2_ndvi_20180521', 'sentinel2_ndvi_20180531',
'sentinel2_ndvi_20180610', 'sentinel2_ndvi_20180620',
'sentinel2_ndvi_20180630', 'sentinel2_ndvi_20180710',
'sentinel2_ndvi_20180720', 'sentinel2_ndvi_20180730',
'sentinel2_ndvi_20180809', 'sentinel2_ndvi_20180819',
'sentinel2_ndvi_20180829', 'sentinel2_ndvi_20180908',
'sentinel2_ndvi_20180918', 'sentinel2_ndvi_20180928',
'sentinel2_ndvi_20181008', 'sentinel2_ndvi_20181018',
'sentinel2_ndvi_20181028', 'sentinel2_ndvi_20181107',
'sentinel2_ndvi_20181117', 'sentinel2_ndvi_20181127',
'sentinel2_ndvi_20181207', 'sentinel2_ndvi_20181217',
'sentinel2_ndvi_20181227'
]

conn = sqlite3.connect(dataset_path)

df_features = pd.read_sql_query("select {} from {}".format(",".join(features_labels), layer_name), conn)
df_labels = pd.read_sql_query("select {} from {}".format(data_field, layer_name), conn)
df_groups = pd.read_sql_query("select {} from {}".format("originfid", layer_name), conn)
splitter = list(GroupKFold(n_splits=cv_folds).split(df_features,
df_labels,
df_groups))
clf = RandomForestClassifier()
clf = GridSearchCV(clf,
cv_parameters,
cv=splitter,
return_train_score=True)

cluster = PBSCluster(cores=12,
memory="60GB")
cluster.adapt(minimum_jobs=5, maximum_jobs=10)
client = Client(cluster)
with joblib.parallel_backend('dask'):
clf.fit(df_features, df_labels)
```

then dask asked me to use client.scatter to deploy data on workers as the following :
```bash
(_CVIterableWrapper(cv=[(array([ 3, 4, .. ... e, False, True)
Consider scattering large objects ahead of time
with client.scatter to reduce scheduler burden and
keep data on workers

future = client.submit(func, big_data) # bad

big_future = client.scatter(big_data) # good
future = client.submit(func, big_future) # good
% (format_bytes(len(b)), s)
```
But if I use the backend like in the tutorial (with scatter) :
```python
with joblib.parallel_backend('dask', scatter=[df_features, df_labels]):
clf.fit(features_values, labels_values)
```
then no workers can be found :
```bash
distributed.core - ERROR - No workers found
Traceback (most recent call last):
File "/.../lib/python3.6/site-packages/distributed/core.py", line 412, in handle_comm
result = await result
File "/.../lib/python3.6/site-packages/distributed/scheduler.py", line 2703, in scatter
raise gen.TimeoutError("No workers found")
tornado.util.TimeoutError: No workers found
```
Any suggestions will be welcome.

**Note** : If I use a smaller dataset, everything works.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.