catboost / catboost/catboost

can't get probability thresholds (catboost.utils.select_threshold) for multi-class (muti-target) models

Open
#2,449 0 comments 0 reactions 0 assignees View on GitHub
feature request
Dominant language
C++
Stars
9.1k
Forks
1.3k
PR merge metrics
No merged PRs in 30d

Description

Problem: select_threshold(model, validation_pool, FNR=0.01) won't work for multi-class problem
catboost version: 1.1.1
Operating System: windows server 2016
CPU: yes
GPU: no

so this video https://youtu.be/usdEWSDisS0?t=3199 London2019 notebook from examples can be found in catboost repository

My code has multiple targets (more than 1 class). It works, however it won't work with getting thresholds for better results as shown on the video above (note the timing in the url)

# I'm using threshold functions from below declarations
from catboost.utils import get_roc_curve
from catboost.utils import get_fpr_curve
from catboost.utils import get_fnr_curve
from catboost.utils import select_threshold
from catboost import CatBoostClassifier

![image](https://github.com/catboost/catboost/assets/20881228/a8b70ebe-1b31-4df8-940f-0ebd2b81a658)

My solution is simply to use 1 class per model (thus ending up with multiple models), but then the whole beauty of catboost is that it can do multiclassification. Also, can I assume that doing it this way I could get better results because individual models can be more complex or it's a wrong intuition? Pls advise.

PS: Btw, suppose I do multiple model files (each has different targets) and will identify needed thresholds for each target. Can I then simply "stick" those models together and use thresholds acquired before? Would the resulting model switch from classification to multi-classification automatically?

Thanks!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.