Lightning-AI / Lightning-AI/torchmetrics
Functional nominal metrics fail for non-zero-based category labels
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 526
- Avg merge
- 6d 11h
- Merged PRs (30d)
- 5
Description
Bug description
The functional nominal association metrics infer the number of categories from the count of unique values, but pass the original category IDs directly to the confusion-matrix implementation. As a result, valid categorical labels that are not zero-based and contiguous can raise a reshape error.
Nominal association should be invariant to renaming categories.
Reproduction
import torch
from torchmetrics.functional.nominal import (
cramers_v,
pearsons_contingency_coefficient,
theils_u,
tschuprows_t,
)
base = torch.tensor([0, 0, 1, 1])
shifted = base + 1
metrics = [
(cramers_v, {"bias_correction": False}),
(pearsons_contingency_coefficient, {}),
(theils_u, {}),
(tschuprows_t, {"bias_correction": False}),
]
for metric, kwargs in metrics:
print(metric(base, base, **kwargs))
print(metric(shifted, shifted, **kwargs))
The first call succeeds for each metric. The relabeled input raises:
RuntimeError: shape '[2, 2]' is invalid for input of size 7
The corresponding matrix APIs are affected as well when columns contain sparse or independently labeled categories.
Expected behavior
The shifted input describes the same two categories and should produce the same association value as the zero-based input. For example, Cramer's V should be tensor(1.) in both cases.
Root cause
The functional paths derive num_classes using the number of unique values, while the classification confusion-matrix updater uses the raw label values as bin indices. With labels 1 and 2, the inferred class count is two, but label 2 creates a bin outside the expected 2 x 2 range.
Normalizing the observed categories to contiguous IDs after logits and NaN preprocessing avoids that mismatch. Stateful metrics that receive an explicit num_classes should keep their existing stable encoding behavior across updates.
Environment
- TorchMetrics: 1.9.0 / current
masterat37a80809 - PyTorch: 2.13.0+cpu
- Python: 3.11.9
- OS: Windows
AI disclosure: I used Codex to help identify and reproduce the behavior, search for existing reports, and draft this issue. I reviewed the reproduction and diagnosis.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the functional nominal entry points in torchmetrics.functional.nominal and the corresponding classification confusion-matrix APIs. Reproduce the shifted-label example, then add regression coverage for sparse or independently labeled categories. Done means relabeled inputs return the same association values without reshape errors while explicit stateful class encodings remain unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- data, machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100