Dask, GridSearchCV, XGBoost on GPU

I am having trouble finding the best practices for running GPU based XGBoost hyper-parameter searches using Dask (RandomizedSearchCV or GridSearchCV from dask-ml). Currently running Windows 7 (don't ask), 128GB RAM, 1x Titan Xp, dual Xeon with combined 16C/32T (8 cores each) with `Python 3.6.5 |Anaconda custom (64-bit)| (default, Mar 29 2018, 13:32:41) [MSC v.1900 64 bit (AMD64)] on win32`.

I use distributed as follows:
```
from dask.distributed import Client
scheduler_address = 'xxx.xxx.xx.xx:8786'
client = Client(scheduler_address)
```

Example dataframe:
```
Index: 526829 entries, 2017-12-01 00:00:00 UTC to 2018-12-04 23:59:00 UTC
Columns: 601 entries, XXX to YYY
dtypes: float32(600), float64(1)
memory usage: 1.2+ GB
```

When using dask.distributed I understand that the GridSearchCV n_jobs parameter is ignored. It seems that jobs are distributed and processed based on the number of threads available. For example, if I run 2 local Worker processes, each with 1 thread, two task processing streams are created. Given that I'm developing on a single machine, and leverage only 1 GPU, I've struggled to find optimal performance and I was wondering whether developers would have an opinion.

Questions:

- Given that certain algorithms like XGB also have an n_jobs equivalent for CPU implementations, how should this be considering in the context of Dask as described above. For example, if you have 1 Worker process running 4 threads, presumably the XGB algorithm running with n_jobs=-1 will be limited to the 4 threads that the Dask Worker process exposes?
- Consider XGBoost GPU: will Dask simply submit multiple XGBoost jobs to the single GPU, or is there some additional scheduling that tasks place?
- Similarly, with a single GPU I often see significant performance differences when running n Worker processes with m threads each, compared to m Worker process with n threads. For example, 1 Worker with 4 threads took 155 seconds to complete the benchmark (fixed number of XGB fits) while 4 Workers with 1 thread each took 97 seconds. Both are faster than 1 Worker 1 Thread (282 seconds). Differences in performance make sense for CPU bound tasks, but I'm unclear of this cause for the GPU instance.

Sorry if this isn't the best place to ask these questions.

The issue names no repository files, tests, or entry points. Start by reviewing the Dask distributed and dask-ml GridSearchCV usage described in the report, then reproduce the stated worker and thread benchmarks if the environment is available. Done would require a decided, documented best-practices answer for CPU and single-GPU scheduling.

Written by the indexing model from the issue text.