catboost / catboost/catboost

difference between full iteration model and model using baseline value

Open
#2,819 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
9.1k
Forks
1.3k
PR merge metrics
No merged PRs in 30d

Description

I cannot understand why the result of full iteration model and result of model using baseline value are different.
I understand 'baseline' parameter of Pool() as the same concept as init_score in LGBM. Isn't it right?
How can i get equal result between full model and model using baseline value? (except using init_model)

Python 3.8.5
sklearn == 0.24.1
catboost == 1.2.2

and My code is as bellow
#######################################################################################

from sklearn.datasets import load_breast_cancer
from sklearn.model selection import train_test_split
import catboost
from catboost import Pool

#######################################################################################
## data load

data = load_breast_cancer()
X, y = data.data, data.target

X_train, X_temp, y_train, y_temp = train_test_split(X, y, train_size = 0.8, stratify = y, random_state = 42)
X_valid, X_test, y_valid, y_test = train_test_split(X_temp, y_temp, train_size = 0.5, stratify = y_temp, random_state = 42)

#######################################################################################
### full iteration model (iteration = 20)

full_catb_model = catboost.CatBoostClassifier(random_state = 1004, iteration = 20, eval_metric = 'AUC', depth = 6, learning_rate = 0.01, max_leaves = 64)
full_catb_model.fit(X = X_train, y = y_train, eval_set = [(X_trian, y_train), (X_valid, y_valid)])

#######################################################################################
### partial iteration model (iteration = 15) -> raw score as baseline -> additional iteration model (iteration = 5)

catb_model = catboost.CatBoostClassifier(random_state = 1004, iteration = 15, eval_metric = 'AUC', depth = 6, learning_rate = 0.01, max_leaves = 64)
catb_model.fit(X = X_train, y = y_train, eval_set = [(X_trian, y_train), (X_valid, y_valid)])

train_raw_score = catb_model.predict(X_train, predicted_type = 'RawFormulaVal')
valid_raw_score = catb_model.predict(X_valid, predicted_type = 'RawFormulaVal')
test_raw_score = catb_model.predict(X_test, predicted_type = 'RawFormulaVal')

train_pool = Pool(X_train, y_train, baseline = train_raw_score)
valid_pool = Pool(X_valid, y_valid, baseline = valid_raw_score)
test_pool = Pool(X_test, y_test, baseline = test_raw_score)

add_catb_model = catboost.CatBoostClassifier(random_state = 1004, iteration = 5, eval_metric = 'AUC', depth = 6, learning_rate = 0.01, max_leaves = 64)
add_catb_model.fit(X = train_pool, eval_set = [train_pool, valid_pool])

#######################################################################################
### compare the predicted result of full model and partial+additional model with baseline value

full_catb_pp = full_catb_model.predict_proba(X_test)[:,1]
add_catb_pp = add_catb_model.predict_proba(test_pool)[:,1]

full_catb_pp[:10] # (result of full model)
add_catb_pp[:10] # (result of partial+additional model with baseline value)
## different!

Contributor guide

Open the contributing guide

Research direction

Reproduce the provided breast-cancer example using CatBoostClassifier, Pool, fit, predict, and predict_proba, correcting only the apparent variable typo as needed. Compare the full 20-iteration model with the 15-plus-5-iteration baseline model and determine whether the differing outputs are expected; document the baseline semantics and the conditions for matching results.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.