Integration of Distributed XGBoost on Modin
- Dominant language
- C++
- Stars
- 28.8k
- Forks
- 8.9k
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 54
Description
Hi XGBoost!
I am from [Modin](https://github.com/modin-project/modin.git) team. Modin provides an efficient distributed DataFrames and has a [distributed implementation of XGBoost.](https://github.com/modin-project/modin/tree/master/modin/experimental/xgboost)
XGBoost already has [support](https://github.com/dmlc/xgboost/pull/6055) of Modin DataFrames, but currently partitions of Modin DataFrame are just transformed to `numpy.array`-s and concatenated to one:
https://github.com/dmlc/xgboost/blob/77f6cf2d134fb7b1fd9af7d80fecb300876015c8/python-package/xgboost/data.py#L244
and possible parallelization between partitions isn't used.
Modin XGBoost is implemented with Ray distribution technology under the hood but support of the other execution engines used in Modin (Dask e.g.) will be added as well. Training and inference happens in parallel between partitions of Modin DataFrame.
Modin team wants to start integration of Modin XGBoost in your repo to have support of distributed Modin DataFrames in the main xgboost package.
The high-level Modin XGBoost documentation can be found [here](https://modin.readthedocs.io/en/latest/modin_xgboost.html). The developer's documentation with implementation details is [here](https://modin.readthedocs.io/en/latest/flow/modin/experimental/xgboost.html).
Are there any requirements for starting the integration?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.