autogluon / autogluon/tabarena
[Meta] Potential integration of ensemble method benchmarking utilities
- Dominant language
- Python
- Stars
- 303
- Forks
- 69
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 49
Description
Hello,
I recently used TabArena to make a benchmark on some basic ensemble models (hard voting, blending, ...). To do this, I made some code that might be useful to other people. I was wondering if it might interest you to integrate some parts directly in TabArena (in which case I would need to better adapt to it).
What I have:
- Ensemble methods that work with TabArena's 3-fold precomputed results for classification
- Soft Voting (+ Median / Max / Min Rules)
- Hard Voting with vote proportion for probabilitic classification
- Blending (Stacking with base learners only trained once and a small meta-dataset)
- Stacking with some caveats
- Uses two sets of 3-fold CV
- Data leakage: Base learners used for the meta-dataset have seen the test set
- (NeuralEnsembler too but it's nearly just copy pasted from [RegularizedNeuralEnsembler](https://github.com/machinelearningnuremberg/RegularizedNeuralEnsemblersBenchmark))
- Some specific TaskEvaluator children
- One that doesn't preprocess wrong probabilities out of existence during prediction (necessary for Hard Voting and methods requiring normalization)
- One that makes a nested 3-fold CV on the test data
- 1/9 of data to train just the ensemble method (like blending)
- 2/9 of data to evaluate
- One specifically for Stacking
- Some multi-family CASH simulation with time budget and iteration budget
- Random Search
- Bandit-based methods (UCB, MaxUCB)
If anything looks interesting, I will need to adapt my code to quite a bit to better integrate it in your project. Notably, I didn't use probability calibration for my code, worked with the 3-fold CV, only ever used the "native" mode, and didn't work on regression.
Have a nice day !
Contributor guide
Research direction
No files or tests are named. Start by reviewing TabArena's existing benchmark and TaskEvaluator structure, then compare the proposed ensemble methods and CASH simulations with current capabilities. Define the integration scope, including classification or regression, calibration, cross-validation, and leakage handling, and identify tests for the accepted behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100