catboost / catboost/catboost

Unexpected behavior with prediction on pools with baseline when loss_function has link function

Open
#1,380 3 comments 0 reactions 0 assignees View on GitHub
bug objectives and metrics
Dominant language
C++
Stars
9.1k
Forks
1.3k
PR merge metrics
No merged PRs in 30d

Description

**Problem**: Unexpected results emerge when predicting on pools that contain a baseline, when the model objective has a link function (e.g., Poisson, Tweedie)
**catboost version**: 0.23.2
**Operating System**: Centos 7
**CPU**: Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
**GPU**: Tesla P100

```python
import numpy as np
import pandas as pd
from catboost import CatBoostRegressor, Pool

base = np.array([1.0, 2.0, 0.5, 0.0, 0.3, 0.0])
multiplier = np.array([0.2, 0.5, 2.0, 0.5, 2.0, 0.2])
train_label = base * multiplier
train_x = pd.DataFrame({'feature': ['a', 'b', 'c', 'b', 'c', 'a']})
def get_regressor():
return CatBoostRegressor(iterations = 3,
loss_function = "Poisson",
random_seed = 125,
depth=4)

# Learn baseline
tr_pool_1 = Pool(data=train_x, label=train_label, cat_features=['feature'])
model1 = get_regressor()
model1.fit(tr_pool_1, verbose=False)

# Learn with the above prediction as baseline
baseline = model1.predict(train_x, prediction_type="RawFormulaVal") # Not sure this prediction_type should be taken

tr_pool_2 = Pool(data=train_x, label=train_label, cat_features=['feature'], baseline=baseline)
model2 = get_regressor()
model2.fit(tr_pool_2, verbose=False)

print("Test cases")
eps = 1e-4

print("Case 1:")
print("Expected: predict on train_x, prediction_type=RawFormulaVal should be np.log(prediction_type=None)")
print("Result:", (np.abs(model2.predict(train_x) - np.exp(model2.predict(train_x, prediction_type="RawFormulaVal"))) <= eps).all())
# Result: True

print("Case 2:")
print("Expected: (RawFormulaVal prediction on tr_pool_2) should be np.log(Default prediction on tr_pool_2)")
print("Result:", (np.abs(model2.predict(tr_pool_2) - np.exp(model2.predict(tr_pool_2, prediction_type="RawFormulaVal"))) <= eps).all())
# Result: False

print("Case 3:")
print("Expected: (model 2 RawFormulaVal prediction on train_x) + (baseline) should be (model 2 RawFormulaVal prediction on tr_pool_2)")
print("Result:", (np.abs(model2.predict(train_x, prediction_type="RawFormulaVal") + baseline - model2.predict(tr_pool_2, prediction_type="RawFormulaVal")) <= eps).all())
# Result: True

print("Case 4:")
print("Expected: (model 2 default prediction on train_x) * exp(baseline) should be (model 2 default prediction on tr_pool_2)")
print("Result:", (np.abs(model2.predict(train_x) * np.exp(baseline) - model2.predict(tr_pool_2)) <= eps).all())
# Result: False
```

Contributor guide

Open the contributing guide

Research direction

Start by running the supplied Python reproduction with CatBoostRegressor, Pool, the Poisson loss function, baselines, and the four prediction cases. Compare default and RawFormulaVal predictions on train_x and tr_pool_2; done means the behavior matches the stated prediction relationships or their intended semantics are clarified.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.