dmlc / dmlc/xgboost

Consistency of min_child_weight parameter

Open
#5,444 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
28.8k
Forks
8.9k
Avg merge
1d 12h
Merged PRs (30d)
54

Description

The `min_child_weight` parameter (default value 1.0) has different effects based on scaling of objective functions. I noticed this when developing a new objective function that had a small Hessian and the tree was not able to grow with default parameters. Objectives like squared error and logistic loss will be regularised very differently as a consequence. For example using logistic loss where the hessian values can be much smaller, it can require a much larger number of training instances to split. In #2483 it is noted that the hessian in the case of logistic loss is proportional to variance, however this is not true of other objectives in general.

This is relevant to the task of finding good default parameters across a range of objectives (#4986).

One obvious solution is normalising all objective functions in some consistent way.

Another solution is deprecating min_child_weight and moving to a parameter like min_child_instances, regularising based on the amount of training data without respect to the objective function.

@trivialfis has also proposed implementing multiclass objective functions via vector leaves, if we do this the hessian will be a vector and it is not obvious how to correctly apply min_child_weight.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.