dmlc / dmlc/xgboost

Order of columns and model accuracy (AUC, F1) on XGBRegressor

Open
#5,224 1 comment 0 reactions 0 assignees View on GitHub
type: bug
Dominant language
C++
Stars
28.8k
Forks
8.9k
Avg merge
1d 12h
Merged PRs (30d)
54

Description

**Context**
I have a very large number of input variables (100s), but only a handful (~50) matters.
I use XGBRegressor to build classification models (xgboost 0.90)
The model's accuracy is quite good (AUC 0.875, F1 0.846)

**Problem**

1. Rebuilding the model using only the important features gives a lower accuracy (AUC 0.872, F1 0.750) than the original model (getting the list of 39 features using `my_model_xgboost.get_booster().get_score`)
2. Rebuilding the model using Shap's most important features gives the same accuracy as the original model. Shap outputs the same 39 columns as above, but in a different order

This does not reproduce on tiny datasets, like the Iris one.

Given that the drop in accuracy is quite significant, I would consider this a bug in the optimisation step of XGBoost. I understand that with equal importance XGBoost would pick the first one, but this does not seem like an equal importance step.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.