Lightning-AI / Lightning-AI/torchmetrics
Incorrect result in computing `MulticlassRecall` macro average when `ignore_index` is specified
@rittik9 is already working on this.
Since Aug 29, 2024.
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 526
- Avg merge
- 6d 11h
- Merged PRs (30d)
- 5
Description
## 🐛 Bug
Specifying the `ignore_index` argument in `MulticlassRecall` leads to incorrect results when computing the macro average.
### To Reproduce
```python
import torch
from torchmetrics.classification.precision_recall import MulticlassRecall
metric = MulticlassRecall(num_classes=2, ignore_index=0, average="macro")
y_true = torch.tensor([0, 0, 1, 1])
y_pred = torch.tensor([
[0.9, 0.1], # Correct
[0.9, 0.1], # Correct
[0.9, 0.1], # Incorrect
[0.1, 0.9], # Correct
])
metric.update(y_pred, y_true)
print(metric.compute()) # Prints 0.25, but I would expect the result to be 0.5
```
### Expected behavior
In the toy example above there are two classes (0 and 1) and two instances of each class. I want to ignore class 0 altogether (hence `ignore_index=0`), so I would expect the first two predictions to be ignored entirely (since they relate to class 0 instances). Of the remaining two predictions only one is correct, so I would expect the recall score to be `0.5`.
However, the actual result is `0.25`. It looks as if the recall score for the `ignore_index` class is simply set to 0 before computing the average, causing the average to be computed as `(0 + 0.5) / 2` instead of `0.5 / 1`.
### Environment
- TorchMetrics version: 1.3.1 (installed via pip)
- Python version: 3.10
- PyTorch version: 2.2.1
- OS: Ubuntu 22.04
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.