dmlc / dmlc/xgboost

Feature Request: add option to use cross-validation to choosing split at each node

Open
#9,560 4 comments 1 reaction 0 assignees View on GitHub
feature-request
Dominant language
C++
Stars
28.8k
Forks
8.9k
Avg merge
1d 12h
Merged PRs (30d)
54

Description

There is a recent paper entitled "Feature Importance in Gradient Boosting Trees with Cross-Validation Feature Selection" (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9140774/) that attempt to address the well known problem of bias towards variables with large cardinalities in feature importance for tree models.

The idea is simple:
- at each node, use cross-validation to select the best split
- i.e. at each node, there are some samples, the split the samples into folds
- for each fold, use the training portion to select metrics (e.g. impurity) as usual, but evaluate the effectiveness on the testing portion
- then for each potential split, average the metrics across folds, then select the best one
- if the average CV metrics is not better than no splitting, then stop splitting

Benefits of this method:

- the paper shows that this can alleviate the bias of feature importance, especially for more efficient feature importance such as gain importance. This is particularly important for real world problems where the number of features is large, and the sample size is also large, so we want to have efficient yet unbiased methods to calculate feature importance, to help feature selection.
- it also provides a natural way to stop growing the tree, i.e. if the CV metrics does not show improvement over not splitting. This should help with preventing overfitting, and also makes hyperparameter tuning easier.

The paper provides reference implementation for simple gradient boosting at https://github.com/aba27059/unbiased_fi_for_gb

It would be great if this can be added to XGBoost.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.