Lightning-AI / Lightning-AI/torchmetrics
NMI/AMI return 0.0 instead of 1.0 for two single-cluster labelings
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 526
- Avg merge
- 6d 11h
- Merged PRs (30d)
- 5
Description
Bug
normalized_mutual_info_score and adjusted_mutual_info_score return 0.0 for two identical single-cluster labelings, where scikit-learn (the documented reference for these metrics) returns 1.0, a perfect match.
Repro
import torch, numpy as np
import sklearn.metrics as skm
import torchmetrics.functional.clustering as tmc
a = np.array([0, 0, 0, 0]); ta = torch.tensor(a)
print(float(tmc.normalized_mutual_info_score(ta, ta)), skm.normalized_mutual_info_score(a, a)) # 0.0 vs 1.0
print(float(tmc.adjusted_mutual_info_score(ta, ta)), skm.adjusted_mutual_info_score(a, a)) # 0.0 vs 1.0
Why
Both labelings have zero entropy, so they trivially agree. scikit-learn treats this limit case as a perfect match and returns 1.0. torchmetrics falls through to the mutual_info == 0 short-circuit and returns 0.0. The asymmetric case (one single cluster, the other not) already returns 0.0 in both libraries and is correct.
Environment: torchmetrics 1.9.0 / current master.
I have a fix ready and will open a PR referencing this issue.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the normalized_mutual_info_score and adjusted_mutual_info_score implementations in torchmetrics.functional.clustering, then run the provided reproduction against the current behavior. Done means identical single-cluster labelings return 1.0 for both metrics while the asymmetric single-cluster case remains 0.0, matching the documented scikit-learn reference.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 75/100