BaseIncrementalSearchCV should use dask_ml.metrics.check_scoring
- Dominant language
- Python
- Stars
- 951
- Forks
- 262
- PR merge metrics
- No merged PRs in 30d
Description
See the discussion at https://github.com/coiled/coiled-examples/pull/1#discussion_r461243952. Ideally we'd be able to pass dask arrays / dataframes to `HyperbandSearchCV.score`. We'd like for `HyperbandSearchCV(estimator, params, scoring="accuracy")` to use dask-ml's accuracy score.
We *probably* want the default `scoring=None` to continue to fall back to scikit-learn estimators. However, there's a pretty strong argument that we could check if the scoring is exactly the same as `RegressorMixin.score` or `ClassificationMixin.score`. In those cases, it may be safe to substitute our scorers which will safely handle Dask collections. If a non-default `score` is used however, we should just call it.
Contributor guide
Research direction
Start by locating BaseIncrementalSearchCV and HyperbandSearchCV, then inspect how HyperbandSearchCV.score currently handles scoring and where dask_ml.metrics.check_scoring is used. Compare the scoring=None, string scorer, and custom non-default score cases described in the issue; done means the intended Dask-collection behavior is covered while custom scores retain their stated behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, scikit-learn
- Domain
- data, machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100