Lightning-AI / Lightning-AI/torchmetrics

Functional nominal metrics raise RuntimeError when category labels do not start at 0

Open
#3,460 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug / fix help wanted
Dominant language
Python
Stars
2.5k
Forks
526
Avg merge
6d 11h
Merged PRs (30d)
5

Description

## 🐛 Bug

Functional nominal metrics (`cramers_v`, `pearsons_contingency_coefficient`, `theils_u`, `tschuprows_t`) raise a `RuntimeError` during confusion matrix calculation when categorical inputs contain non-zero-based or non-contiguous category labels (e.g., labels `[1, 2]` or `[5, 6]`).

### To Reproduce

Code sample

```python
import torch
from torchmetrics.functional.nominal import cramers_v

preds = torch.tensor([5, 6, 5, 6, 5, 6, 5, 6])
target = torch.tensor([5, 6, 5, 5, 6, 6, 5, 6])

# Raises RuntimeError: shape '[2, 2]' is invalid for input of size 19
cramers_v(preds, target)
```

Environment

- TorchMetrics version: 1.9.0 (latest master)
- Python & PyTorch Version: Python 3.10+, PyTorch 2.x
- Any other relevant information such as OS: All platforms (Linux, macOS, Windows)

### Additional context

The functional API infers `num_classes = len(torch.cat([preds, target]).unique())` but passes the unmapped raw values to `_multiclass_confusion_matrix_update`, which expects values strictly in `[0, num_classes - 1]`. Any values outside this range cause invalid bin indexing and reshape failures. Additionally, `_compute_chi_squared` performs in-place float addition on integer `confmat` tensors, causing type cast issues. Remapping categorical labels to contiguous 0-based indices before updating the confusion matrix resolves this issue.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the functional nominal metric entry points for cramers_v, pearsons_contingency_coefficient, theils_u, and tschuprows_t, then trace their use of _multiclass_confusion_matrix_update and _compute_chi_squared. Reproduce the supplied labels [5, 6] and verify that all four metrics calculate successfully without reshape or tensor type-cast errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.