dmlc / dmlc/xgboost

[R] bug in creation of stratified folds

Open
#5,670 11 comments 0 reactions 0 assignees View on GitHub
cross-validation type: bug
Dominant language
C++
Stars
28.8k
Forks
8.9k
Avg merge
1d 12h
Merged PRs (30d)
54

Description

Dear xgboost developers,
this is a follow-up to my issue #4509. I have upgrade to xgboost 1.0.0.2 and have checked the behaviour of the creation of stratified folds.
The first problem I have reported in #4509 has been fixed with #4631. Thank you very much for this.
The second problem is still there. Please have a look at the following code:

library(xgboost)

# evaluation function that prints number of actual positive instances
numpos <- function(pred, dtrain){
truth <- as.numeric(xgboost::getinfo(dtrain, 'label'))
print(sum(truth))
return(list(metric = 'numpos', value = sum(truth)))
}

features_train = matrix(C<-(1:8),nrow=8, ncol=1)
targets_train = c(0,0,0,0,1,1,1,1)
dtrain <- xgb.DMatrix(features_train, label = targets_train)
param <- list(objective = "binary:logistic")

set.seed(314159)
xgb.cv(data=dtrain, nrounds=1, nfold=2,params=param, stratified = TRUE, feval = numpos, verbose=2)

When I run this code, I get the following output
[1] 3
[1] 1
[1] 1
[1] 3
This means that I get one fold with 3 positive labels and 1 fold with 1 positive label. In stratified sampling, I would expect 2 positive labels in every fold, i.e the output
[1] 2
[1] 2
[1] 2
[1] 2

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.