dmlc / dmlc/xgboost

Different results from using scale_pos_weight vs oversampling

Open
#11,516 14 comments 0 reactions 0 assignees View on GitHub
feature-request
Dominant language
C++
Stars
28.8k
Forks
8.9k
Avg merge
1d 12h
Merged PRs (30d)
54

Description

I get different results when I take a toy dataset and set `scale_pos_weight = 10` versus taking that same dataset and oversampling the positive class by a factor of 10. I assumed that these two approaches would yield the same results. I can't seem to find any documentation about the difference between `scale_pos_weight` and oversampling. Is there anything I'm missing?

NB: I get a similar difference when I run SparkXGBClassifier with a weight column that's 10x for the positive class versus when I run with `scale_pos_weight`.

I have a synthetic, toy dataset generated as follows:

```
training_rows = []

for i in range(1000000):
row = {
## A correlates with our label.
"A": (-1 if i % 2 == 0 else 1) * random.random(),
## B is an integer similar to A - I'd like to see if xgboost thinks it's more important.
"B": round((-100 if i % 2 == 1 else 100) * random.random()),
## C is a constant and shouldn't be important
"C": 5,
## D is entirely random and shouldn't be important.
"D": random.random(),
"E": True if random.random() > 0.8 else False,
"F": 1 if i % 2 == 0 else 0}
training_rows.append(row)
```

And the model is trained as follows:

```
xgb_params = {'features_col': 'features',
'label_col': 'F',
'validation_indicator_col': 'E',
'weight_col': 'weight',
'eval_metric': 'aucpr',
'early_stopping_rounds': 10,
'num_workers': sc.defaultParallelism,
'tree_method': 'hist',
'n_estimators': 100,
'seed': 123,
}

classifier = SparkXGBClassifier(**xgb_params)

xgb_model = classifier.fit(assembled_train_val)

print(xgb_model.get_booster().best_iteration)
print(xgb_model.get_booster().best_score)
xgb_model.get_booster().feature_names = features
x = xgb_model.get_feature_importances(importance_type='gain')

{k: v for k, v in sorted(x.items(), key=lambda item: item[1], reverse=True)}
```

For the scale_pos_weight I replace 'weight_col' with scale_pos_weight and for oversampling I remove both these parameters.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.