dmlc / dmlc/xgboost

Changing the order of rows in a toy dataset yields dramatically different predictions for XGBRanker

Open
#10,025 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
28.8k
Forks
8.9k
Avg merge
1d 12h
Merged PRs (30d)
54

Description

Consider the simple script below:

```python
import pandas as pd
import numpy as np
import xgboost as xgb

def fit_and_print_ranker_dump_and_predictions(X, y, qids):
model = xgb.XGBRanker(objective='rank:ndcg', seed=0, random_state=0)
model.fit(X, y, qid=qids)
print(model.get_booster().get_dump(dump_format='text')[0])
print(pd.Series(model.predict(X), index=X.index))

X = pd.DataFrame([
[7.5, 3.5],
[1.5, 4.0],
[7.0, 2.0],
[3.0, 8.0],
[4.5, 3.5],
[8.0, 8.0],
], columns=['feature1', 'feature2'])

y = pd.Series(
[0,
0,
1, # relevant item in group 0
1, # relevant item in group 1
0,
0])

qids = [0, 0, 0, 1, 1, 1]

print("XGBranker on original dataset yields the tree shown below, followed by the predictions shown below:")
fit_and_print_ranker_dump_and_predictions(X, y, qids)

# now permute the rows
permutation = [2, 1, 0, 3, 4, 5]
X_permuted = X.iloc[permutation, :]
y_permuted = y[permutation]
print("XGBranker on permuted dataset yields the tree shown below, followed by the predictions shown below:")
fit_and_print_ranker_dump_and_predictions(X_permuted, y_permuted, qids)
```

On my machine (M1 Macbook), I get:

```
XGBranker on original dataset yields the tree shown below, followed by the predictions shown below:
0:[feature1<5.75] yes=1,no=2,missing=1
1:leaf=0.026957728
2:leaf=-0.0239090417

0 0.356739
1 0.643589
2 0.356739
3 0.643589
4 0.582729
5 0.356739
dtype: float32
XGBranker on permuted dataset yields the tree shown below, followed by the predictions shown below:
0:[feature1<5.75] yes=1,no=2,missing=1
1:leaf=-0.0151539575
2:leaf=0.0168569554

2 0.796365
1 -0.174696
0 0.796365
3 0.264934
4 0.375095
5 0.686204
dtype: float32
```

Note that the rankers completely disagree. In the first dataset, the highest ranked item in the first group is item 1 (score of 0.64), and in the permuted dataset, item 1 is the lowest ranked item (score of -0.17).

On a real world dataset with hundreds of thousands of rows, we also found that permuting a single pair of rows can lead to massive differences in the resulting rankers and predictions. Also, we found that swapping two feature values (out of more than 100 features) for only a single row can also lead to massive differences. And also, we found that training the same model with the same data on two different machines can also lead to massive differences. Curiously, the degree of difference is approximately the same for all three cases, i.e. if we look at various metrics, and create baseline_model, model_with_two_swapped_rows, model_with_two_swapped_values_in_a_single_row, and model_trained_on_other_machine, any pairwise comparison of the two rows shows nearly the same degree of impact to the recs (recs include same top rec, same top 3 recs, same top 10 recs).

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.