xgb.cv's auc score is not matching with cross_val_score when `colsample_bytree` is other than 1
- Dominant language
- C++
- Stars
- 28.8k
- Forks
- 8.9k
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 54
Description
I am working on highly imbalanced dataset. During hyperparameter tuning, I found that if colssample_bytree is set to value other than 1, then cross_val_score from sklearn package is not matching with the auc score obtained from xgb.cv.
xgb.cv code:
```
# creating kfolds
kfolds = StratifiedKFold(n_splits = 5, shuffle = True, random_state = 16)
# creating model object and using it for xgb.cv
xgb0 = XGBClassifier(objective= 'binary:logistic', n_estimators =2, colsample_bytree = 0.6,
random_state =16, n_jobs = -1, eval_metric = 'auc')
params = xgb0.get_params()
xg_train = xgb.DMatrix(X_train_p.values, label = y_train.values)
cv_result = xgb.cv(params, xg_train, num_boost_round=2, folds = kfolds, metrics = 'auc', early_stopping_rounds = 50,
as_pandas = True, seed = 16,stratified=True, shuffle = True)
print(cv_result['test-auc-mean'].values[-1])
```
This results in test-auc-mean of 0.91706
cross_val_score code:
```
cv_score = cross_val_score(xgb0, X_train_p, y_train, cv = kfolds, n_jobs = -1, scoring = 'roc_auc')
cv_score.mean()
```
This results in test-auc-mean of 0.8994
I don't understand this huge difference between these and as I already indicated if colsample_bytree is set to 1 then there is no difference between scores. Also this difference between auc scores increase significantly as we decrease colsample_bytree.
Can someone help me in understanding why this is happening. Thanks
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.