Lightning-AI / Lightning-AI/torchmetrics

ClasswiseWrapper silently drops values for invalid labels

Open Beginner friendly
#3,472 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2.5k
Forks
526
Avg merge
6d 11h
Merged PRs (30d)
5

Description

Bug

ClasswiseWrapper converts a metric tensor into a dictionary with zip(self.labels, x). If the number of labels does not match the number of metric values, zip silently truncates the longer input. Duplicate labels also overwrite earlier dictionary entries. Both cases can make class metrics disappear without any error.

Reproduction

import torch

from torchmetrics.classification import MulticlassAccuracy
from torchmetrics.wrappers import ClasswiseWrapper

metric = ClasswiseWrapper(
    MulticlassAccuracy(num_classes=3, average=None),
    labels=["cat", "dog"],
)

result = metric(
    torch.tensor([0, 1, 2]),
    torch.tensor([0, 1, 2]),
)
print(result)

The wrapped metric produces three class values, but the result contains only two:

{"multiclassaccuracy_cat": tensor(1.), "multiclassaccuracy_dog": tensor(1.)}

Likewise, labels such as ["cat", "cat", "dog"] produce only two dictionary keys because the second cat overwrites the first.

Expected behavior

Custom labels should define a one-to-one mapping to the wrapped metric values. Mismatched label counts and duplicate labels should raise a clear ValueError instead of silently losing results.

Environment

  • TorchMetrics: current master at e63e92a6
  • PyTorch: 2.13.0
  • Python: 3.12.13
  • macOS

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the ClasswiseWrapper implementation and reproduce the mismatch and duplicate-label cases from the issue using the shown PyTorch metrics. Add coverage for both invalid inputs and verify that each raises a clear ValueError rather than silently dropping or overwriting metric values.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.