RFC: Revisit Default Hyperparameters for Tree Learning in XGBoost

**Related discussions:** XGBoost issue **#4986** (“Revisiting default parameter settings?”) and issue **#6034** (“Add default ranges for hyperparameter tuning”). ([github.com](https://github.com/dmlc/xgboost/issues/4986?utm_source=chatgpt.com))

## Summary

This RFC proposes a modest update to XGBoost’s default tree-learning configuration to improve out-of-the-box robustness and to better align the shipped defaults with published tunability results.

Current XGBoost parameter docs list `eta=0.3`, `max_depth=6`, `min_child_weight=1`, `subsample=1`, and `colsample_bytree=1`. The Python API docs still list `num_boost_round=10` for both `xgboost.train` and `xgboost.cv`, while the current `master` branch source still sets `DEFAULT_N_ESTIMATORS = 100` for the scikit-learn wrapper. ([xgboost.readthedocs.io](https://xgboost.readthedocs.io/en/stable/parameter.html))

### Proposed defaults

| Parameter | Current | Proposed |
|---|---:|---:|
| `eta` | 0.3 | 0.1 |
| `max_depth` | 6 | ~~10~~ 6 |
| `min_child_weight` | 1 | 2 |
| `subsample` | 1.0 | 0.8 |
| `colsample_bytree` | 1.0 | 0.8 |
| `xgboost.train(..., num_boost_round=...)` | 10 | 300 |
| `xgboost.cv(..., num_boost_round=...)` | 10 | 300 |
| sklearn fallback `n_estimators` | 100 | 300 |

### Parameters not changed

- `gamma = 0`
- `colsample_bylevel = 1`
- `reg_lambda = 1`
- `reg_alpha = 0` ([xgboost.readthedocs.io](https://xgboost.readthedocs.io/en/stable/parameter.html))

## Motivation

The strongest academic evidence on XGBoost hyperparameter tunability remains Probst, Boulesteix, and Bischl (2019). Their cross-dataset study found that XGBoost has meaningful tunability under package defaults and materially lower tunability after replacing them with optimized defaults. The paper also states explicitly that, in XGBoost, the joint gain of tuning `nrounds` and `eta` is relatively low because the two are highly connected: when `nrounds` is set higher, `eta` should generally be set lower, and vice versa. ([jmlr.org](https://jmlr.org/papers/volume20/18-444/18-444.pdf))

That same paper reports learned XGBoost defaults far from the current shipped defaults. The optimal-default values they report include very small `eta`, `subsample` below 1, `colsample_bytree` below 1, `min_child_weight` above 1, `max_depth` well above 6, and `nrounds` in the thousands rather than tens or hundreds. The exact reported values vary by metric, but the direction is consistent: smaller learning rate, larger round budget, moderate row/feature subsampling, somewhat deeper trees, and slightly more conservative leaf growth. ([jmlr.org](https://jmlr.org/papers/volume20/18-444/18-444.pdf?utm_source=chatgpt.com))

The broader boosting literature points in the same direction. Friedman’s original gradient boosting paper states that shrinkage introduces two coupled regularization parameters, the learning rate `v` and the number of components `M`, and that decreasing `v` increases the best value of `M`. XGBoost’s own parameter docs describe `eta` as step-size shrinkage that makes the process more conservative, and the Python API supports `early_stopping_rounds` directly. ([cse.cuhk.edu.hk](https://www.cse.cuhk.edu.hk/irwin.king/_media/presentations/2001_greedy_function_approximation_a_gradient_boosting_machine.pdf))

## Proposal

### 1. Update core tree-learning defaults

Change the default tree-learning parameters to:

- `eta = 0.1`
- `max_depth = 10`
- `min_child_weight = 2`
- `subsample = 0.8`
- `colsample_bytree = 0.8` ([xgboost.readthedocs.io](https://xgboost.readthedocs.io/en/stable/parameter.html))

### 2. Align Python-facing training budgets

Set the Python-facing default training budgets to:

- `xgboost.train(..., num_boost_round=300, ...)`
- `xgboost.cv(..., num_boost_round=300, ...)`
- sklearn fallback `DEFAULT_N_ESTIMATORS = 300` ([xgboost.readthedocs.io](https://xgboost.readthedocs.io/en/latest/python/python_api.html))

This alignment removes the current discrepancy between the native Python learning API and the scikit-learn wrapper. There is little published justification for exposing a much smaller default training budget in one Python interface than in another when both ultimately control the same sequence of additive tree updates. The tunability literature instead argues that learning rate and training budget are tightly coupled and should be considered together. ([jmlr.org](https://jmlr.org/papers/volume20/18-444/18-444.pdf))

### 3. Strengthen documentation around early stopping

Documentation and examples should more explicitly pair smaller `eta` with:

- larger round ceilings, and
- early stopping when validation data are available. ([xgboost.readthedocs.io](https://xgboost.readthedocs.io/en/latest/python/python_api.html))

## Rationale by parameter

### `eta: 0.3 → 0.1`

The current docs list `eta=0.3` and describe it as step-size shrinkage used to make boosting more conservative. Probst et al. report much smaller learned defaults for XGBoost than 0.3, and identify `eta` as one of the most tunable XGBoost parameters. Friedman’s shrinkage analysis also supports smaller learning rates when training budget permits. A move from 0.3 to 0.1 follows both the XGBoost documentation and the academic literature while avoiding a more disruptive jump to extremely small values. ([xgboost.readthedocs.io](https://xgboost.readthedocs.io/en/stable/parameter.html))

### `max_depth: 6 → 10`

The current docs list `max_depth=6` and warn that deeper trees are more complex, more likely to overfit, and can consume memory aggressively. Probst et al. nevertheless report learned XGBoost defaults of 13, 14, and 11 across the metrics they study, all materially above the current default of 6. A default of 10 moves the shipped baseline closer to that region without fully matching the larger learned depths from the benchmark study. ([xgboost.readthedocs.io](https://xgboost.readthedocs.io/en/stable/parameter.html))

### `min_child_weight: 1 → 2`

The current docs list `min_child_weight=1` and describe larger values as more conservative. Probst et al. report learned defaults for XGBoost in the neighborhood of 1.3 to 2.1, which supports a modest upward shift rather than a large one. A default of 2 is therefore consistent with both the documented semantics and the benchmark-based optimal defaults. ([xgboost.readthedocs.io](https://xgboost.readthedocs.io/en/stable/parameter.html))

### `subsample: 1.0 → 0.8`

The docs state that `subsample` is the row subsampling ratio and that subsampling helps prevent overfitting. Probst et al. report learned XGBoost defaults consistently below 1.0, roughly in the high-0.8 range depending on metric. A default of 0.8 is slightly more conservative than the learned defaults, but still very close to the region favored by the cross-dataset study. ([xgboost.readthedocs.io](https://xgboost.readthedocs.io/en/stable/parameter.html))

### `colsample_bytree: 1.0 → 0.8`

The docs state that `colsample_bytree` is the fraction of columns subsampled when constructing each tree, and that the `colsample_by*` parameters work cumulatively. Probst et al. report learned defaults for `colsample_bytree` around 0.71 to 0.75. A default of 0.8 moves clearly in the literature-supported direction while avoiding compounded regularization, since this RFC leaves `colsample_bylevel` unchanged. ([xgboost.readthedocs.io](https://xgboost.readthedocs.io/en/stable/parameter.html))

## Rationale for aligned default round counts

The strongest justification for increasing and aligning default round counts is internal consistency with the lower learning rate. The Python API currently defaults to `num_boost_round=10`, while the sklearn wrapper source still falls back to 100 estimators. At the same time, the literature says smaller learning rates require more iterations, and the XGBoost-specific tunability paper states directly that `nrounds` and `eta` are highly connected. ([xgboost.readthedocs.io](https://xgboost.readthedocs.io/en/latest/python/python_api.html))

A default of 300 is justified as a practical aligned budget. Reducing `eta` from 0.3 to 0.1 lowers the per-round step size by a factor of three. Raising the sklearn fallback from 100 to 300 preserves that rough scaling. Aligning `xgboost.train` and `xgboost.cv` to the same 300-round default then removes the current API inconsistency. The literature supports the coupling between learning rate and rounds, even though it does not provide a universal exact scaling constant. ([cse.cuhk.edu.hk](https://www.cse.cuhk.edu.hk/irwin.king/_media/presentations/2001_greedy_function_approximation_a_gradient_boosting_machine.pdf))

## Why other defaults are left unchanged

`gamma`, `reg_lambda`, and `reg_alpha` are left unchanged because the evidence for changing them is weaker and less consistent than the evidence for changing `eta`, training budget, moderate subsampling, `min_child_weight`, and depth. `colsample_bylevel` is left unchanged because the docs emphasize that the `colsample_by*` parameters are cumulative, so changing both `colsample_bytree` and `colsample_bylevel` simultaneously would stack the regularization. ([xgboost.readthedocs.io](https://xgboost.readthedocs.io/en/stable/parameter.html))

## Expected benefits

The expected effect is a more internally consistent and more robust out-of-the-box tree learner: smaller boosting steps, a larger and aligned default training budget, moderate row and feature randomness, a slightly higher child-weight threshold, and deeper baseline interaction capacity. This proposed default set is also materially closer to the regions favored by the cross-dataset tunability literature than the current documented defaults. ([jmlr.org](https://jmlr.org/papers/volume20/18-444/18-444.pdf?utm_source=chatgpt.com))

## Risks and compatibility

The largest risk is increased training time from larger default training budgets. That increase is intentional: lowering `eta` without raising default round counts would leave the defaults internally inconsistent. A second risk is larger models and higher memory use, especially with `max_depth=10`; the parameter docs explicitly warn that deep trees can consume memory aggressively. A third risk is that enabling `subsample` and `colsample_bytree` below 1 introduces stochasticity into workflows that currently rely on fully deterministic default sampling settings. ([xgboost.readthedocs.io](https://xgboost.readthedocs.io/en/stable/parameter.html))

## Suggested release-note wording

Tree-learning defaults in XGBoost have been updated to improve robustness and internal consistency out of the box. The default learning rate was reduced, default depth and split conservatism were rebalanced, moderate row and feature subsampling were enabled by default, and Python-facing default training budgets were aligned upward to match the lower learning rate. Users comparing results across releases should account for the larger default training budget and should consider early stopping when validation data are available. ([xgboost.readthedocs.io](https://xgboost.readthedocs.io/en/latest/python/python_api.html))

## References

- Probst, Boulesteix, and Bischl (2019), *Tunability: Importance of Hyperparameters of Machine Learning Algorithms*. ([jmlr.org](https://jmlr.org/papers/volume20/18-444/18-444.pdf?utm_source=chatgpt.com))
- Friedman (2001), *Greedy Function Approximation: A Gradient Boosting Machine*. ([cse.cuhk.edu.hk](https://www.cse.cuhk.edu.hk/irwin.king/_media/presentations/2001_greedy_function_approximation_a_gradient_boosting_machine.pdf?utm_source=chatgpt.com))
- XGBoost parameter documentation. ([xgboost.readthedocs.io](https://xgboost.readthedocs.io/en/stable/parameter.html?utm_source=chatgpt.com))
- XGBoost Python API documentation. ([xgboost.readthedocs.io](https://xgboost.readthedocs.io/en/latest/python/python_api.html?utm_source=chatgpt.com))
- XGBoost issue #4986. ([github.com](https://github.com/dmlc/xgboost/issues/4986?utm_source=chatgpt.com))
- XGBoost issue #6034. ([github.com](https://github.com/dmlc/xgboost/issues/6034?utm_source=chatgpt.com))

Start with the xgboost.train and xgboost.cv entry points, the sklearn wrapper, and the DEFAULT_N_ESTIMATORS constant mentioned in the issue. Compare their current defaults with the parameter and Python API documentation, then review the proposed early-stopping guidance and release-note wording. Done means the default changes and documentation direction are agreed consistently across these interfaces.

Written by the indexing model from the issue text.