dask / dask/dask-ml

PCA+pipeline+GridSearchCV error

Open
#629 13 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
951
Forks
262
PR merge metrics
No merged PRs in 30d

Description

There seems to be an issue with sklearn PCA+pipeline and dask_ml gridsearchCV. Please see my example below. Apologies if I am totally missing something.

Relevant Versions:

```
dask: 2.12.0
dask_ml: 1.2.0
sklearn: 0.22.2.post1
```

### Minimal Example:

The following code shows that
* Pipeline+PCA+Sklearn_GridSearchCV works
* Pipeline+SelectKBest+Dask_GridSearchCV works
* Pipeline+PCA+Dask_GridSearchCV fails

```python
from dask.distributed import Client, LocalCluster
from dask_ml.model_selection import GridSearchCV as dask_GS
from sklearn.model_selection import GridSearchCV as sk_GS
from sklearn.datasets import make_multilabel_classification
from sklearn import svm
from sklearn.pipeline import Pipeline
from sklearn.decomposition import PCA
from sklearn.feature_selection import SelectKBest, f_classif
import numpy as np

cluster = LocalCluster(n_workers=2,threads_per_worker=2)
client = Client(cluster)

X, Y = make_multilabel_classification(n_classes=12, n_labels=1,n_features=271,
n_samples=1200,
random_state=1)
Y = Y.sum(axis=1)

N_FEATURES_OPTIONS_pca = np.arange(10)[1::3]
N_FEATURES_OPTIONS_sel = np.arange(10)[1::3]
Cs = [1,10,100.]
gammas = [.001,0.01]

pca = PCA(iterated_power='auto')
selection = SelectKBest(f_classif)
svc = svm.SVC()

pipe1 = Pipeline([
('reduce_dim', selection),
('classify', svc)])

pipe2 = Pipeline([
('reduce_dim', pca),
('classify', svc)])

param_grid1 = [{'reduce_dim__k': N_FEATURES_OPTIONS_sel,
'classify__C': Cs,
'classify__gamma': gammas}]
param_grid2 = [{'reduce_dim__n_components': N_FEATURES_OPTIONS_pca,
'classify__C': Cs,
'classify__gamma': gammas}]

#Sklearn Gridsearch with PCA pipeline
sk_clf = sk_GS(pipe2, param_grid2,cv=3,scoring='f1_macro',refit=True)
sk_clf.fit(X,Y)
print(sk_clf.best_score_)

#Dask Gridsearch with SelectKbest
dask_clf1 = dask_GS(pipe1, param_grid1,cv=3,scheduler=client,scoring='f1_macro',refit=True)
dask_clf1.fit(X,Y)
print(dask_clf1.best_score_)

#Dask Gridsearch with PCA
dask_clf2 = dask_GS(pipe2, param_grid2,cv=3,scheduler=client,scoring='f1_macro',refit=True)
dask_clf2.fit(X,Y)
print(dask_clf2.best_score_)

```

This results in several core dump files and the following error:

Results In:

```
0.18358307544807187
0.16009973534039232
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
('score-f44a4381cb4779b9d45ba2c0ba7c2a72', 15, 1) has failed... retrying
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
in
55 #Dask Gridsearch with PCA
56 dask_clf2 = dask_GS(pipe2, param_grid2,cv=3,scheduler=client,scoring='f1_macro',refit=True)
---> 57 dask_clf2.fit(X,Y)
58 print(dask_clf2.best_score_)

/opt/conda/envs/py_geo/lib/python3.7/site-packages/dask_ml/model_selection/_search.py in fit(self, X, y, groups, **fit_params)
1255 else:
1256 logger.warning("{} has failed... retrying".format(future.key))
-> 1257 future.retry()
1258 ac.add(future)
1259

/opt/conda/envs/py_geo/lib/python3.7/site-packages/distributed/client.py in retry(self, **kwargs)
307 Client.retry
308 """
--> 309 return self.client.retry([self], **kwargs)
310
311 def cancelled(self):

/opt/conda/envs/py_geo/lib/python3.7/site-packages/distributed/client.py in retry(self, futures, asynchronous)
2139 futures: list of Futures
2140 """
-> 2141 return self.sync(self._retry, futures, asynchronous=asynchronous)
2142
2143 @gen.coroutine

/opt/conda/envs/py_geo/lib/python3.7/site-packages/distributed/client.py in sync(self, func, asynchronous, callback_timeout, *args, **kwargs)
778 else:
779 return sync(
--> 780 self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
781 )
782

/opt/conda/envs/py_geo/lib/python3.7/site-packages/distributed/utils.py in sync(loop, func, callback_timeout, *args, **kwargs)
346 if error[0]:
347 typ, exc, tb = error[0]
--> 348 raise exc.with_traceback(tb)
349 else:
350 return result[0]

/opt/conda/envs/py_geo/lib/python3.7/site-packages/distributed/utils.py in f()
330 if callback_timeout is not None:
331 future = asyncio.wait_for(future, callback_timeout)
--> 332 result[0] = yield future
333 except Exception as exc:
334 error[0] = sys.exc_info()

/opt/conda/envs/py_geo/lib/python3.7/site-packages/tornado/gen.py in run(self)
733
734 try:
--> 735 value = future.result()
736 except Exception:
737 exc_info = sys.exc_info()

/opt/conda/envs/py_geo/lib/python3.7/site-packages/distributed/client.py in _retry(self, futures)
2128 response = await self.scheduler.retry(keys=keys, client=self.id)
2129 for key in response:
-> 2130 st = self.futures[key]
2131 st.retry()
2132

KeyError: "('pca-fit-transform-f44a4381cb4779b9d45ba2c0ba7c2a72', 0, 1)"
```

Dask Distributed worker / scheduler logs

```
{'Scheduler': 'distributed.scheduler - INFO - Clear task state\n'
'distributed.scheduler - INFO - Scheduler at: '
'tcp://127.0.0.1:41727\n'
'distributed.scheduler - INFO - dashboard at: '
'127.0.0.1:8787\n'
'distributed.scheduler - INFO - Register worker \n"
'distributed.scheduler - INFO - Starting worker compute stream, '
'tcp://127.0.0.1:35216\n'
'distributed.scheduler - INFO - Register worker \n"
'distributed.scheduler - INFO - Starting worker compute stream, '
'tcp://127.0.0.1:44716\n'
'distributed.scheduler - INFO - Receive client connection: '
'Client-ee79b350-6fec-11ea-878c-0cc47a4279e3\n'
'distributed.scheduler - INFO - Remove worker \n"
'distributed.scheduler - INFO - Register worker \n"
'distributed.scheduler - INFO - Starting worker compute stream, '
'tcp://127.0.0.1:38830\n'
'distributed.scheduler - INFO - Remove worker \n"
'distributed.scheduler - INFO - Register worker \n"
'distributed.scheduler - INFO - Starting worker compute stream, '
'tcp://127.0.0.1:43574\n'
'distributed.scheduler - INFO - Remove worker \n"
'distributed.scheduler - INFO - Remove worker \n"
'distributed.scheduler - INFO - Lost all workers\n'
'distributed.scheduler - INFO - Client '
'Client-ee79b350-6fec-11ea-878c-0cc47a4279e3 requests to retry 1 '
'keys\n'
'distributed.scheduler - INFO - Register worker \n"
'distributed.scheduler - INFO - Starting worker compute stream, '
'tcp://127.0.0.1:33853\n'
'distributed.scheduler - INFO - Register worker \n"
'distributed.scheduler - INFO - Starting worker compute stream, '
'tcp://127.0.0.1:33057\n'
'distributed.scheduler - INFO - Remove worker \n"
'distributed.scheduler - INFO - Register worker \n"
'distributed.scheduler - INFO - Starting worker compute stream, '
'tcp://127.0.0.1:33579\n'
'distributed.scheduler - INFO - Remove worker \n"
'distributed.scheduler - INFO - Register worker \n"
'distributed.scheduler - INFO - Starting worker compute stream, '
'tcp://127.0.0.1:33304\n'
'distributed.scheduler - INFO - Remove worker \n"
'distributed.scheduler - INFO - Register worker \n"
'distributed.scheduler - INFO - Starting worker compute stream, '
'tcp://127.0.0.1:40951',
'tcp://127.0.0.1:33304': 'distributed.worker - INFO - Start worker '
'at: tcp://127.0.0.1:33304\n'
'distributed.worker - INFO - Listening '
'to: tcp://127.0.0.1:33304\n'
'distributed.worker - INFO - Waiting to connect '
'to: tcp://127.0.0.1:41727\n'
'distributed.worker - INFO - '
'-------------------------------------------------\n'
'distributed.worker - INFO - '
'Threads: 2\n'
'distributed.worker - INFO - '
'Memory: 6.50 GB\n'
'distributed.worker - INFO - Local Directory: '
'/project/cper_neon_aop/neon_temporal/dask-worker-space/worker-nswgxjil\n'
'distributed.worker - INFO - '
'-------------------------------------------------\n'
'distributed.worker - INFO - Registered '
'to: tcp://127.0.0.1:41727\n'
'distributed.worker - INFO - '
'-------------------------------------------------',
'tcp://127.0.0.1:40951': 'distributed.worker - INFO - Start worker '
'at: tcp://127.0.0.1:40951\n'
'distributed.worker - INFO - Listening '
'to: tcp://127.0.0.1:40951\n'
'distributed.worker - INFO - Waiting to connect '
'to: tcp://127.0.0.1:41727\n'
'distributed.worker - INFO - '
'-------------------------------------------------\n'
'distributed.worker - INFO - '
'Threads: 2\n'
'distributed.worker - INFO - '
'Memory: 6.50 GB\n'
'distributed.worker - INFO - Local Directory: '
'/project/cper_neon_aop/neon_temporal/dask-worker-space/worker-31u54x2o\n'
'distributed.worker - INFO - '
'-------------------------------------------------\n'
'distributed.worker - INFO - Registered '
'to: tcp://127.0.0.1:41727\n'
'distributed.worker - INFO - '
'-------------------------------------------------'}
```

Results of gdb on core dump file:

```
Core was generated by `/opt/conda/envs/py_geo/bin/python -c from multiprocessing.forkserver import mai'.
Program terminated with signal 11, Segmentation fault.
```

Contributor guide

Open the contributing guide

Research direction

Start by running the minimal example with the reported Dask, dask-ml, and scikit-learn versions, then inspect dask_ml/model_selection/_search.py at the future.retry call and the distributed worker and scheduler logs. Done means PCA inside a pipeline completes under dask_ml GridSearchCV without worker restarts or the missing-future KeyError, while the SelectKBest and scikit-learn comparisons remain valid.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, scikit-learn
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.