Lightning-AI / Lightning-AI/torchmetrics
ClasswiseWrapper silently drops values for invalid labels
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 526
- Avg merge
- 6d 11h
- Merged PRs (30d)
- 5
Description
Bug
ClasswiseWrapper converts a metric tensor into a dictionary with zip(self.labels, x). If the number of labels does not match the number of metric values, zip silently truncates the longer input. Duplicate labels also overwrite earlier dictionary entries. Both cases can make class metrics disappear without any error.
Reproduction
import torch
from torchmetrics.classification import MulticlassAccuracy
from torchmetrics.wrappers import ClasswiseWrapper
metric = ClasswiseWrapper(
MulticlassAccuracy(num_classes=3, average=None),
labels=["cat", "dog"],
)
result = metric(
torch.tensor([0, 1, 2]),
torch.tensor([0, 1, 2]),
)
print(result)
The wrapped metric produces three class values, but the result contains only two:
{"multiclassaccuracy_cat": tensor(1.), "multiclassaccuracy_dog": tensor(1.)}
Likewise, labels such as ["cat", "cat", "dog"] produce only two dictionary keys because the second cat overwrites the first.
Expected behavior
Custom labels should define a one-to-one mapping to the wrapped metric values. Mismatched label counts and duplicate labels should raise a clear ValueError instead of silently losing results.
Environment
- TorchMetrics: current
masterate63e92a6 - PyTorch: 2.13.0
- Python: 3.12.13
- macOS
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the ClasswiseWrapper implementation and reproduce the mismatch and duplicate-label cases from the issue using the shown PyTorch metrics. Add coverage for both invalid inputs and verify that each raises a clear ValueError rather than silently dropping or overwriting metric values.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100