dmlc / dmlc/xgboost

RandomForest in XGBoost

Open
#6,608 1 comment 0 reactions 0 assignees View on GitHub
doc
Dominant language
C++
Stars
28.8k
Forks
8.9k
Avg merge
1d 12h
Merged PRs (30d)
54

Description

The [Caveats](https://xgboost.readthedocs.io/en/latest/tutorials/rf.html#caveats) in docs for running RandomForest in XGBoost are saying:

> XGBoost uses 2nd order approximation to the objective function. This can lead to results that differ from a random forest implementation that uses the exact value of the objective function.

It seems to me when you do eg:

```
xgb = XGBClassifier(
n_estimators=300,
max_depth=3,
objective="binary:logistic",
eval_metric="logloss",
use_label_encoder=False,
)
xgb
```
```
XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,
colsample_bynode=1, colsample_bytree=1, eval_metric='logloss',
gamma=0, gpu_id=-1, importance_type='gain',
interaction_constraints='', learning_rate=0.300000012,
max_delta_step=0, max_depth=3, min_child_weight=1, missing=nan,
monotone_constraints='()', n_estimators=300, n_jobs=12,
num_parallel_tree=1, objective='binary:logistic', random_state=0,
reg_alpha=0, reg_lambda=1, scale_pos_weight=1, subsample=1,
tree_method='exact', use_label_encoder=False,
validate_parameters=1, verbosity=None)
```
second order approximation is not the main difference from sklearn's result. The main difference seem to be the objective function: `"gini"` for `sklearn` and `"logloss"` (?) for `XGBoost` (please correct me if I am wrong)

And the choice of objective function, not the order of approximation, will affect probability calibration curves:

![image](https://user-images.githubusercontent.com/8139370/104700773-b8287e00-5725-11eb-95f8-f80b8ec2fba8.png)

![image](https://user-images.githubusercontent.com/8139370/104700819-caa2b780-5725-11eb-8d37-acba5ebbe2dd.png)

with calibration curves for XGBoost with `booster="gbtree"` being perfectly (as expected) calibrated for bigger datasets.

So my proposal here is to add this to the docs (assuming I am right)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.