Lightning-AI / Lightning-AI/torchmetrics

NMI/AMI return 0.0 instead of 1.0 for two single-cluster labelings

Open Beginner friendly
#3,469 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2.5k
Forks
526
Avg merge
6d 11h
Merged PRs (30d)
5

Description

Bug

normalized_mutual_info_score and adjusted_mutual_info_score return 0.0 for two identical single-cluster labelings, where scikit-learn (the documented reference for these metrics) returns 1.0, a perfect match.

Repro

import torch, numpy as np
import sklearn.metrics as skm
import torchmetrics.functional.clustering as tmc

a = np.array([0, 0, 0, 0]); ta = torch.tensor(a)
print(float(tmc.normalized_mutual_info_score(ta, ta)), skm.normalized_mutual_info_score(a, a))  # 0.0 vs 1.0
print(float(tmc.adjusted_mutual_info_score(ta, ta)), skm.adjusted_mutual_info_score(a, a))        # 0.0 vs 1.0

Why

Both labelings have zero entropy, so they trivially agree. scikit-learn treats this limit case as a perfect match and returns 1.0. torchmetrics falls through to the mutual_info == 0 short-circuit and returns 0.0. The asymmetric case (one single cluster, the other not) already returns 0.0 in both libraries and is correct.

Environment: torchmetrics 1.9.0 / current master.

I have a fix ready and will open a PR referencing this issue.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the normalized_mutual_info_score and adjusted_mutual_info_score implementations in torchmetrics.functional.clustering, then run the provided reproduction against the current behavior. Done means identical single-cluster labelings return 1.0 for both metrics while the asymmetric single-cluster case remains 0.0, matching the documented scikit-learn reference.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
75/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.