autogluon / autogluon/autogluon

Per-model calibration

Open
#1,667 0 comments 1 reaction 0 assignees View on GitHub
enhancement module: tabular priority: 2
Dominant language
Python
Stars
10.7k
Forks
1.2k
Avg merge
21h 29m
Merged PRs (30d)
57

Description

Apply calibration to each model in the ensemble to potentially improve results more than simply calibrating the final weighted ensemble. Has potential to improve metrics such as ROC-AUC, which is currently not improved via final weighted ensemble calibration.

Steps:
1. Try per-model calibration on non-bagged setting -> Temperature scale each model after it trains
2. Try to calibrate a bagged ensemble -> Either just use out-of-fold predictions for calibration or calibrate each child individually.
3. Try to calibrate end-to-end multi-layer stack ensemble -> Calibrate each bagged ensemble of the stack + weighted ensemble.

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by locating the model-training and ensemble-calibration paths, then compare the three proposed experiments: non-bagged temperature scaling, bagged calibration, and multi-layer stack calibration. Done means evaluating whether per-model calibration improves metrics such as ROC-AUC over final weighted-ensemble calibration.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.