Dask-GridSearchCv is 3-4x slow compared to Sklearn-GridSearchCv (using MLPClassifier with SGD optimizer)

**Background:**

I've been doing a research project which requires conducting some experiments to compare runtime baselines for data parallel and task parallel hyperparameter tuning techniques for Neural Network training.

Two of these experiments involved comparing sklearn's GridSearchCV against dask's GridSearchCV implementation when run on a single node. Since Dask's implementation is more or less a wrapper over Sklearn's implementation, I was expecting very similar results. The setup for this experiment was:

- Single node experiment
- 8 CPUs
- 32 GB RAM
- Dataset: https://archive.ics.uci.edu/ml/datasets/HIGGS (Size 8gb, 11M records, 28 numerical features)
- Fixed Neural Net Architecture
- Sklearn version: 0.19.1
- Dask-ML version: 0.4.1

**Code and relevant files**
Sklearn GridSearchCV python code, Dask GridSearchCV python code, their log files and generated pickle files are present in the attached zip file.

From here-on, I refer to **sklearn's GridSearchCV experiment as exp3** and **dask's GridSearchCV experiment as exp4**. This is just to make things easier to understand, and consistent with my uploaded code for reproducibility.

exp3 and exp4 are exactly similar except just one import statement.
`from sklearn.model_selection import GridSearchCV` in exp3
gets replaced by
`dask_ml.model_selection import GridSearchCV` in exp4

**Hyperparameter grid**
`param_grid = {'mlpclassifier__batch_size': [128, 256, 512, 1024], 'mlpclassifier__hidden_layer_sizes':[(16,16)], 'mlpclassifier__solver':['lbfgs','sgd'], 'mlpclassifier__learning_rate_init':[0.0001, 0.001, 0.01, 0.1], 'mlpclassifier__max_iter': [50], 'mlpclassifier__nesterovs_momentum': [False], 'mlpclassifier__activation': ['logistic'], 'mlpclassifier__alpha': [0.0001, 0.001, 0.01, 0.1], 'mlpclassifier__momentum': [0], 'mlpclassifier__shuffle': [False], 'mlpclassifier__random_state': [1]}`

**Additional info that may be relevant**
My goal for both of these experiments was to maximize CPU usage while still being able to use all cores in the machine. First, for exp3, I tried to use all 8 cores for parallel model training (setting "n_jobs=8" and "pre_dispatch=8" in GridCV params). Because of the dataset size and may be internal creation of temporary arrays, it was not possible, I faced memory overflow errors when I tried running GridSearchCV using sklearn's implementation on all 8 cores. However, I was able use 4 cores ("n_jobs"=4 and "pre_dispatch"=4 in GridCV params) without blowing up memory, and hence could train 4 models in parallel on the full dataset. Since Dask's GridSearchCV calls Sklearn's GridSearchCV internally, I expected to get the same behavior for exp4, and I did get it.

**The actual issue**
As can be seen in the param grid, there are a total of 128 different models that will be trained for both exp3 and exp4. Out of those 128 models, 64 will be for "lbfgs" and 64 will be for "sgd" as the optimization (solver) method. Since DaskML's implementation calls Sklearn's GridSearchCV implementation, I was hoping to observe similar training times for each model configuration across exp3 and exp4. As shown in the "Observing Issue" and "Reproducing Issue" section below, I was able to get very similar training runtimes for lbfgs as the solver, whereas sgd was 3-4x slower for dask. I went over the dask's source code but couldn't locate why this is happening. Exp3 and exp4 literally have the same code apart from a single import statement change, and a missing "pre_dispatch" argument in dask's GridSearchCV (as it is not provided for tuning).

**Observing Issue**

- Use python 2
- load pickle files as:
`exp3 = pkl.load(open("exp3_output_sklearn_pipelined_PD4_NJ4_03_52_03_03_2018.pkl", "rb" ))
exp4 = pkl.load(open("exp4_output_dask_MLP_singleNode_pipelined_NJ4_21_03_03_03_2018.pkl", "rb" ))
`
- The above pickle files have the entire "clf.cv_results_" dictionaries
- do `exp3.keys()` or `exp4.keys()` to get all the results as a sanity check.
- do
`print(exp3['mean_fit_time'][2]) #one of the 64 configurations for LBFGS, exp3 (sklearn)
print(exp4['mean_fit_time'][2]) #one of the 64 configurations for LBFGS, exp4 (dask)
print(exp3['mean_fit_time'][3]) #one of the 64 configurations for SGD, exp3 (sklearn)
print(exp4['mean_fit_time'][3]) #one of the 64 configurations for SGD, exp4 (dask)
`

- You should get:
806.5635731220245
786.9201691150665
2003.3694651126862
8084.313671112061

- Clearly there is 3-4x slowdown for SGD but not for LBFGS, which is strange. This is more or less consistent across all 64 model configurations. Please feel free to verify.

**Reproducing Issue**
Run the python files after downloading the HIGGS dataset and extracting in the same directory as the code. Make sure versions are consistent. Feel free to reduce the number of hyperparameter combinations to save wait time.

[exp3 and exp4.zip](https://github.com/dask/dask-ml/files/1802095/exp3.and.exp4.zip)