Lightning-AI / Lightning-AI/torchmetrics

AUROC reported wrong value when called differently

Open
#1,989 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug / fix help wanted v1.0.x
Dominant language
Python
Stars
2.5k
Forks
526
Avg merge
6d 11h
Merged PRs (30d)
5

Description

🐛 Bug

AUROC calculation is different when being called differently.

To Reproduce

Copy the following code and run.

Note that when .update() is called, values that are greater than 1.0 will be scaled using the sigmoid activation, values that is less than 1.0 will not be scaled. This result in the compute step to compute with the wrong input values. However, directly calling the auroc will not rescale the value

from torchmetrics import AUROC
from torch import tensor

pp = tensor([tensor([1.2]), tensor([0.9678]), tensor([1.7])])
tt = tensor([tensor([1]), tensor([0]), tensor([1])])

auroc = AUROC(num_classes=1, task="binary")
print("Forward:", auroc(pp, tt))


for p, t in zip(pp, tt):
    auroc.update(p, t)
    print(auroc.preds, auroc.target)

print("Update and Compute:", auroc.compute())

Expected behavior

The reported AUROC metrics for both cases should be the same.

Environment
  • TorchMetrics 1.0.3
  • Python 3.9.16
  • PyTorch 1.13.1

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the supplied Python reproduction with TorchMetrics 1.0.3 and compare AUROC's direct-call path with its update/compute path. Trace the AUROC forward, update, and compute entry points; done means both paths report the same value for the reproduction.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.