dask / dask/dask-ml

Integration with dask-xgboost.

Open
#906 4 comments 2 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
951
Forks
262
PR merge metrics
No merged PRs in 30d

Description

Hi, I want to work on the integration between dask-ml and dask-xgboost. Specifically, I want to implement `GridSearchCV` and `RandomizedGridSearchCV` for estimators like dask-xgboost, which are already distributed by themselves. I tried to look into the existing grid search cv implementation in dask-ml, which seems to be written by manipulating the dask task graph instead of using documented public API. I would like to ask for some guidance before I go ahead and try to implement yet another grid search cv. Some questions are:
- What's the best way to handle algorithms that are already aware of dask and can run on distributed systems?
- Why is the `GridSearchCV` written in this way instead of using public API? Should I be using it in a downstream project? If so is there a good place to learn more about it?
- Any chance of upstreaming the integration into dask-ml if I were able to implement it? It will be useful not only for XGBoost, but also for other similar projects like LightGBM.

Related:
https://github.com/dask/dask-ml/issues/833
https://github.com/dmlc/xgboost/issues/5676
https://github.com/dask/dask-ml/issues/758

Contributor guide

Open the contributing guide

Research direction

Start by reading the existing GridSearchCV implementation and the related issues #833 and #758, then review the linked XGBoost issue #5676. The issue needs agreement on how distributed estimators should be integrated before implementation can begin; done would require an agreed design and an upstream implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.