dmlc / dmlc/xgboost

Integration of Distributed XGBoost on Modin

Open
#7,094 2 comments 0 reactions 0 assignees View on GitHub
feature-request
Dominant language
C++
Stars
28.8k
Forks
8.9k
Avg merge
1d 12h
Merged PRs (30d)
54

Description

Hi XGBoost!

I am from [Modin](https://github.com/modin-project/modin.git) team. Modin provides an efficient distributed DataFrames and has a [distributed implementation of XGBoost.](https://github.com/modin-project/modin/tree/master/modin/experimental/xgboost)

XGBoost already has [support](https://github.com/dmlc/xgboost/pull/6055) of Modin DataFrames, but currently partitions of Modin DataFrame are just transformed to `numpy.array`-s and concatenated to one:
https://github.com/dmlc/xgboost/blob/77f6cf2d134fb7b1fd9af7d80fecb300876015c8/python-package/xgboost/data.py#L244
and possible parallelization between partitions isn't used.

Modin XGBoost is implemented with Ray distribution technology under the hood but support of the other execution engines used in Modin (Dask e.g.) will be added as well. Training and inference happens in parallel between partitions of Modin DataFrame.

Modin team wants to start integration of Modin XGBoost in your repo to have support of distributed Modin DataFrames in the main xgboost package.

The high-level Modin XGBoost documentation can be found [here](https://modin.readthedocs.io/en/latest/modin_xgboost.html). The developer's documentation with implementation details is [here](https://modin.readthedocs.io/en/latest/flow/modin/experimental/xgboost.html).

Are there any requirements for starting the integration?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.