RandomForest in XGBoost
- Dominant language
- C++
- Stars
- 28.8k
- Forks
- 8.9k
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 54
Description
The [Caveats](https://xgboost.readthedocs.io/en/latest/tutorials/rf.html#caveats) in docs for running RandomForest in XGBoost are saying:
> XGBoost uses 2nd order approximation to the objective function. This can lead to results that differ from a random forest implementation that uses the exact value of the objective function.
It seems to me when you do eg:
```
xgb = XGBClassifier(
n_estimators=300,
max_depth=3,
objective="binary:logistic",
eval_metric="logloss",
use_label_encoder=False,
)
xgb
```
```
XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,
colsample_bynode=1, colsample_bytree=1, eval_metric='logloss',
gamma=0, gpu_id=-1, importance_type='gain',
interaction_constraints='', learning_rate=0.300000012,
max_delta_step=0, max_depth=3, min_child_weight=1, missing=nan,
monotone_constraints='()', n_estimators=300, n_jobs=12,
num_parallel_tree=1, objective='binary:logistic', random_state=0,
reg_alpha=0, reg_lambda=1, scale_pos_weight=1, subsample=1,
tree_method='exact', use_label_encoder=False,
validate_parameters=1, verbosity=None)
```
second order approximation is not the main difference from sklearn's result. The main difference seem to be the objective function: `"gini"` for `sklearn` and `"logloss"` (?) for `XGBoost` (please correct me if I am wrong)
And the choice of objective function, not the order of approximation, will affect probability calibration curves:


with calibration curves for XGBoost with `booster="gbtree"` being perfectly (as expected) calibrated for bigger datasets.
So my proposal here is to add this to the docs (assuming I am right)
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.