dmlc / dmlc/xgboost

[RFC] Deprecate the `Slice` method of `DMatrix`.

Open
#5,556 8 comments 0 reactions 0 assignees View on GitHub
cross-validation status: RFC
Dominant language
C++
Stars
28.8k
Forks
8.9k
Avg merge
1d 12h
Merged PRs (30d)
54

Description

This is a proposal that we delegate data slicing to external libraries.

## Motivation

`DMatrix` is an internal data interface, used for all XGBoost algorithm. The `DMatrix` object has a few different representation, based on which some more variants can be generated at runtime. Currently we have 3 different base representation for `DMatrix`, namely a CSR like `SparsePage`, a Ellpack sparse matrix for storing GPU histogram index, and lastly external memory batched support for above two in-memory formats.

The `Slice` method for `DMatrix` is mostly used for early stopping where we need to split data for folds. But the implementation is rather insufficient for many use cases as it supports only the `SparsePage` and without shuffling support, not to mention distributed environment support. Also when used with ranking, it doesn't maintain the `group_ptr_`, which is delegated to a higher level Python wrapper script, resulting in an inconsistent support status for different language bindings.

## Moving forward

It has became clear that the cross validation implementation in xgboost is here to stay due to callback feature. So a better way to reduce complexity in `DMatrix` is to seek possible implementation of cv without having to slice up `DMatrix`. It might be possible that we can slice up input data instead.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.