Is learning_curve fully supported by Dask?
- Dominant language
- Python
- Stars
- 951
- Forks
- 262
- PR merge metrics
- No merged PRs in 30d
Description
The problem is fully described [here](https://stackoverflow.com/questions/55953522/is-sklearn-learning-curve-function-supported-by-dask):
For 2 levels of parallelism like in GridSearchCV or RandomSearchCV problems, Dask provides its own implementation. Now for [learning_curve](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.learning_curve.html), there is also such 2 level of parallelism. Can it be simply handled like this?
```
estimator = RandomForestRegressor(n_jobs=-1, **rf_params)
with joblib.parallel_backend('dask', scatter=[X,Y]):
train_sizes, train_scores, test_scores = learning_curve(estimator, X, Y, cv=cv, n_jobs=-1, train_sizes=train_sizes)
```
If you take a look at the [source code](https://github.com/scikit-learn/scikit-learn/blob/7b136e9/sklearn/model_selection/_validation.py#L1242-L1246) for the parallel part, you will see:
```
out = parallel(delayed(_fit_and_score)(
clone(estimator), X, y, scorer, train, test, verbose,
parameters=None, fit_params=None, return_train_score=True,
error_score=error_score)
for train, test in train_test_proportions)
```
If a Dask `Client` is created with `processes=False` (multi-threading parallelism), then Dask shows a warning saying that 2 levels of parallelism have been found and therefore the "inner" one is forced to have `n_jobs=1`. But this is not the case for multiprocessing parallelism.
In particular in the case of RandomForests, what should be the value of n_jobs for the `estimator` and for the `learning_curve` call?
(since joblib 0.12.2, RandomForests backend is not enforced to be threading, see PR [here](https://github.com/scikit-learn/scikit-learn/pull/11741/files#diff-65dc1c804f310acfd90f5ea83286065c))
Contributor guide
Research direction
Start with sklearn.model_selection.learning_curve and the referenced sklearn/model_selection/_validation.py parallel block. Reproduce the warning using a Dask Client with processes=False and compare estimator and learning_curve n_jobs settings for RandomForestRegressor under the dask joblib backend. Done means the supported configuration and multiprocessing behavior are clarified in tests or documentation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, scikit-learn
- Domain
- distributed-systems, machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100