[Inconsistency]: Fbeta metric lacks is_multilabel and advanced average parameters
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.8k
- Forks
- 726
- Avg merge
- 5d 21h
- Merged PRs (30d)
- 5
Description
🚀 Feature Request / Bug Report
Currently, the Fbeta metric in PyTorch-Ignite behaves inconsistently compared to its underlying components, Precision and Recall. While Precision and Recall support the is_multilabel flag and multiple average modes (macro, micro, weighted, samples), Fbeta only supports a boolean average and lacks the is_multilabel flag entirely.
Motivation
When a user wants to compute a multi-label F1-score, they expect to be able to do:
# This currently fails with TypeError: Fbeta() got an unexpected keyword argument 'is_multilabel'
f1 = Fbeta(beta=1.0, is_multilabel=True)
Currently, the user is forced to manually instantiate \Precision\ and \Recall\ with \verage=False\ and \is_multilabel=True\ and pass them to \Fbeta\. I noticed that even in historical issues (e.g., #2444), maintainers have provided this manual workaround instead of having a native API path in \Fbeta\. This adds unnecessary boilerplate and creates an inconsistent experience across classification metrics.
### Proposed Implementation
I would like to resolve this inconsistency by:
1. Adding \is_multilabel: bool = False\ to the \Fbeta\ signature.
2. Passing \is_multilabel\ down to the internally created \Precision\ and \Recall\ objects.
3. Updating the \verage\ parameter handling to eventually support more granular modes (micro, weighted, etc.) or documenting why it differs from \Precision/Recall\.
I have already verified this limitation locally and am prepared to submit a PR that includes:
- Updated \Fbeta\ signature and logic.
- Comprehensive unit tests for multi-label F-beta scores (which are currently missing in \ est_fbeta.py\).
@vfdev-5 I'd love to hear your thoughts on aligning this API!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Fbeta implementation and test_fbeta.py, then compare its parameters with the existing Precision and Recall APIs. Confirm the intended average modes with maintainers; done should include is_multilabel support and unit tests for multi-label F-beta behavior, with the average behavior clarified or covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning, testing
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100