Lightning-AI / Lightning-AI/torchmetrics

`BinaryPrecisionRecallCurve` computes wrong value if used with logits, even though the docstring says this is supported

Open
#2,329 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug / fix help wanted v1.3.x
Dominant language
Python
Stars
2.5k
Forks
526
Avg merge
6d 11h
Merged PRs (30d)
5

Description

## 🐛 Bug

The object oriented `BinaryPrecisionRecallCurve` can compute substantially incorrect values if logits are passed as predictions instead of probabilities, even though the docstring says this is ok. The underlying functional version `binary_precision_recall_curve` seems to work correctly in both cases. Both versions attempt to convert the logits to probabilities by passing them through a sigmoid *if any values are outside of the range [0, 1]*. In the object oriented case this condition is incorrectly checked independently for each batch, rather than for the metric as a whole. Consequently some batches may have sigmoid applied to their scores, while others do not, resulting in an incorrect curve for the dataset as a whole.

### To Reproduce

```py
import torch
import torchmetrics

b1 = torchmetrics.AUROC(task="binary")
b2 = torchmetrics.AUROC(task="binary")

score = torch.tensor([-1.0, 0.0, 1.0])
label = torch.tensor([1.0, 0.0, 1.0])

b1.update(score, label) # pass all three score/label pairs in as a single batch

for s, l in zip(score, label):
b2.update(torch.tensor([s]), torch.tensor([l])) # pass score/label pairs in one at a time

assert b1.compute().item() == 0.5
assert b2.compute().item() == 1.0
```

### Expected behavior

I would expect both AUCs to equal 0.5, as computed with scikit-learn using `sklearn.metrics.roc_auc_score(label, score)`.

### Environment

- TorchMetrics version (and how you installed TM, e.g. `conda`, `pip`, build from source): 1.3.0.post0
- Python & PyTorch Version (e.g., 1.0): 3.10.13
- Any other relevant information such as OS (e.g., Linux): MacOS

### Additional context

The bug is on line https://github.com/Lightning-AI/torchmetrics/blob/master/src/torchmetrics/classification/precision_recall_curve.py#L165 -- as it is currently written the function `_binary_precision_recall_curve_format` should only be called on the full dataset, not on individual batches. Otherwise the behavior is wrong if some batches have all scores in the range [0, 1] but other batches do not.

Some possible solutions are: (1) update the docs not to allow for logits in the object oriented interface, since the behavior is correct for probabilities; (2) don't try to automatically infer whether to apply sigmoid -- my choice, but would be a breaking change (3) refactor `_binary_precision_recall_curve_format` and accept that if any values are found which require sigmoid, then all values from past batches need to have sigmoid applied (this would be tricky in the case where thresholds are specified because the scores are not kept around).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read src/torchmetrics/classification/precision_recall_curve.py around line 165 and inspect _binary_precision_recall_curve_format, then run the supplied reproduction with batched and one-item updates. Done means the object-oriented and functional behavior is consistent for logits across batching, including the documented behavior and threshold-related cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.