dmlc / dmlc/xgboost

xgb.cv's auc score is not matching with cross_val_score when `colsample_bytree` is other than 1

Open
#5,448 5 comments 0 reactions 0 assignees View on GitHub
cross-validation type: bug
Dominant language
C++
Stars
28.8k
Forks
8.9k
Avg merge
1d 12h
Merged PRs (30d)
54

Description

I am working on highly imbalanced dataset. During hyperparameter tuning, I found that if colssample_bytree is set to value other than 1, then cross_val_score from sklearn package is not matching with the auc score obtained from xgb.cv.

xgb.cv code:
```
# creating kfolds
kfolds = StratifiedKFold(n_splits = 5, shuffle = True, random_state = 16)

# creating model object and using it for xgb.cv
xgb0 = XGBClassifier(objective= 'binary:logistic', n_estimators =2, colsample_bytree = 0.6,
random_state =16, n_jobs = -1, eval_metric = 'auc')
params = xgb0.get_params()
xg_train = xgb.DMatrix(X_train_p.values, label = y_train.values)
cv_result = xgb.cv(params, xg_train, num_boost_round=2, folds = kfolds, metrics = 'auc', early_stopping_rounds = 50,
as_pandas = True, seed = 16,stratified=True, shuffle = True)
print(cv_result['test-auc-mean'].values[-1])
```
This results in test-auc-mean of 0.91706

cross_val_score code:

```
cv_score = cross_val_score(xgb0, X_train_p, y_train, cv = kfolds, n_jobs = -1, scoring = 'roc_auc')
cv_score.mean()
```
This results in test-auc-mean of 0.8994

I don't understand this huge difference between these and as I already indicated if colsample_bytree is set to 1 then there is no difference between scores. Also this difference between auc scores increase significantly as we decrease colsample_bytree.

Can someone help me in understanding why this is happening. Thanks

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.