Lightning-AI / Lightning-AI/torchmetrics

Functional nominal metrics fail for non-zero-based category labels

Open
#3,446 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2.5k
Forks
526
Avg merge
6d 11h
Merged PRs (30d)
5

Description

Bug description

The functional nominal association metrics infer the number of categories from the count of unique values, but pass the original category IDs directly to the confusion-matrix implementation. As a result, valid categorical labels that are not zero-based and contiguous can raise a reshape error.

Nominal association should be invariant to renaming categories.

Reproduction

import torch

from torchmetrics.functional.nominal import (
    cramers_v,
    pearsons_contingency_coefficient,
    theils_u,
    tschuprows_t,
)

base = torch.tensor([0, 0, 1, 1])
shifted = base + 1

metrics = [
    (cramers_v, {"bias_correction": False}),
    (pearsons_contingency_coefficient, {}),
    (theils_u, {}),
    (tschuprows_t, {"bias_correction": False}),
]

for metric, kwargs in metrics:
    print(metric(base, base, **kwargs))
    print(metric(shifted, shifted, **kwargs))

The first call succeeds for each metric. The relabeled input raises:

RuntimeError: shape '[2, 2]' is invalid for input of size 7

The corresponding matrix APIs are affected as well when columns contain sparse or independently labeled categories.

Expected behavior

The shifted input describes the same two categories and should produce the same association value as the zero-based input. For example, Cramer's V should be tensor(1.) in both cases.

Root cause

The functional paths derive num_classes using the number of unique values, while the classification confusion-matrix updater uses the raw label values as bin indices. With labels 1 and 2, the inferred class count is two, but label 2 creates a bin outside the expected 2 x 2 range.

Normalizing the observed categories to contiguous IDs after logits and NaN preprocessing avoids that mismatch. Stateful metrics that receive an explicit num_classes should keep their existing stable encoding behavior across updates.

Environment

  • TorchMetrics: 1.9.0 / current master at 37a80809
  • PyTorch: 2.13.0+cpu
  • Python: 3.11.9
  • OS: Windows

AI disclosure: I used Codex to help identify and reproduce the behavior, search for existing reports, and draft this issue. I reviewed the reproduction and diagnosis.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the functional nominal entry points in torchmetrics.functional.nominal and the corresponding classification confusion-matrix APIs. Reproduce the shifted-label example, then add regression coverage for sparse or independently labeled categories. Done means relabeled inputs return the same association values without reshape errors while explicit stateful class encodings remain unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
data, machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.