dmlc / dmlc/xgboost

Structure of multi-output trees

Open
#12,277 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
28.8k
Forks
8.9k
Avg merge
1d 12h
Merged PRs (30d)
54

Description

In multi-output trees, leafs should have one value for each output, but if inspecting the trees, they appear to have more - guess multiplied by the depth or something like that. Example:
```python
import numpy as np
rng = np.random.default_rng(seed=123)
X = rng.standard_normal(size=(100,10))
y = rng.integers(5, size=X.shape[0])

import xgboost as xgb
model = xgb.train(
dtrain=xgb.DMatrix(X, y),
num_boost_round=1,
params={
"objective": "multi:softprob",
"max_depth": 2,
"multi_strategy": "multi_output_tree",
"num_class": 5,
},
)

import json
json.loads(
model.save_raw(raw_format="json")
)["learner"]["gradient_booster"]["model"]["trees"][-1]["leaf_weights"]
```
```
[0.13317914,
-0.12695906,
0.12011204,
-0.16318427,
0.10858103,
-0.0880465,
-0.083756156,
-0.13272737,
0.26806664,
0.04055968,
-0.13248305,
0.22173962,
-0.13717051,
-0.017899837,
-0.046972796,
0.01588887,
-0.03370758,
0.033632353,
0.0655839,
-0.057034157]
```

Has 20 values, yet the raw predictions output something with 5 values.

How do those numbers relate to the raw/margin predictions?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the provided Python reproducer, focusing on model.save_raw(raw_format="json") and the final tree's leaf_weights. Compare those values with the model's raw and margin predictions for the same inputs, tracing how multi-output tree leaves are interpreted. Done means explaining the 20-versus-5 value relationship or identifying the inconsistent serialization or prediction behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.