autogluon / autogluon/autogluon

Add more custom eval metric options to the MM module.

Open
#2,797 3 comments 0 reactions 0 assignees View on GitHub
API & Doc enhancement module: multimodal priority: 1
Dominant language
Python
Stars
10.7k
Forks
1.2k
Avg merge
21h 29m
Merged PRs (30d)
57

Description

Adding a custom eval metric to the MM predictor is almost impossible.
It's a link to the torchmetrics and there it's a way doing a custom torchmetric funtions, which then should be added to the predictor. I don't think many will succeed doing this without use many hours of work. The torchmetric has also limited default metrics to use.
How about adding additional other possible metrics, such as the one from the tabular predictor function which is compatible with common codes, even if it's not equal as fast and optimized as the torchmetric.

Contributor guide

Open the contributing guide

Research direction

Start with the MM predictor's custom evaluation metric handling and its torchmetrics integration. Compare the available tabular predictor metrics with the MM module's current options, then define which additional metrics and custom-function path are in scope. Done means the selected metrics can be supplied to and used by the MM predictor.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.