difference between full iteration model and model using baseline value
- Dominant language
- C++
- Stars
- 9.1k
- Forks
- 1.3k
- PR merge metrics
- No merged PRs in 30d
Description
I cannot understand why the result of full iteration model and result of model using baseline value are different.
I understand 'baseline' parameter of Pool() as the same concept as init_score in LGBM. Isn't it right?
How can i get equal result between full model and model using baseline value? (except using init_model)
Python 3.8.5
sklearn == 0.24.1
catboost == 1.2.2
and My code is as bellow
#######################################################################################
from sklearn.datasets import load_breast_cancer
from sklearn.model selection import train_test_split
import catboost
from catboost import Pool
#######################################################################################
## data load
data = load_breast_cancer()
X, y = data.data, data.target
X_train, X_temp, y_train, y_temp = train_test_split(X, y, train_size = 0.8, stratify = y, random_state = 42)
X_valid, X_test, y_valid, y_test = train_test_split(X_temp, y_temp, train_size = 0.5, stratify = y_temp, random_state = 42)
#######################################################################################
### full iteration model (iteration = 20)
full_catb_model = catboost.CatBoostClassifier(random_state = 1004, iteration = 20, eval_metric = 'AUC', depth = 6, learning_rate = 0.01, max_leaves = 64)
full_catb_model.fit(X = X_train, y = y_train, eval_set = [(X_trian, y_train), (X_valid, y_valid)])
#######################################################################################
### partial iteration model (iteration = 15) -> raw score as baseline -> additional iteration model (iteration = 5)
catb_model = catboost.CatBoostClassifier(random_state = 1004, iteration = 15, eval_metric = 'AUC', depth = 6, learning_rate = 0.01, max_leaves = 64)
catb_model.fit(X = X_train, y = y_train, eval_set = [(X_trian, y_train), (X_valid, y_valid)])
train_raw_score = catb_model.predict(X_train, predicted_type = 'RawFormulaVal')
valid_raw_score = catb_model.predict(X_valid, predicted_type = 'RawFormulaVal')
test_raw_score = catb_model.predict(X_test, predicted_type = 'RawFormulaVal')
train_pool = Pool(X_train, y_train, baseline = train_raw_score)
valid_pool = Pool(X_valid, y_valid, baseline = valid_raw_score)
test_pool = Pool(X_test, y_test, baseline = test_raw_score)
add_catb_model = catboost.CatBoostClassifier(random_state = 1004, iteration = 5, eval_metric = 'AUC', depth = 6, learning_rate = 0.01, max_leaves = 64)
add_catb_model.fit(X = train_pool, eval_set = [train_pool, valid_pool])
#######################################################################################
### compare the predicted result of full model and partial+additional model with baseline value
full_catb_pp = full_catb_model.predict_proba(X_test)[:,1]
add_catb_pp = add_catb_model.predict_proba(test_pool)[:,1]
full_catb_pp[:10] # (result of full model)
add_catb_pp[:10] # (result of partial+additional model with baseline value)
## different!
Contributor guide
Research direction
Reproduce the provided breast-cancer example using CatBoostClassifier, Pool, fit, predict, and predict_proba, correcting only the apparent variable typo as needed. Compare the full 20-iteration model with the 15-plus-5-iteration baseline model and determine whether the differing outputs are expected; document the baseline semantics and the conditions for matching results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100