Lightning-AI / Lightning-AI/torchmetrics
`BinaryPrecisionRecallCurve` computes wrong value if used with logits, even though the docstring says this is supported
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 526
- Avg merge
- 6d 11h
- Merged PRs (30d)
- 5
Description
## 🐛 Bug
The object oriented `BinaryPrecisionRecallCurve` can compute substantially incorrect values if logits are passed as predictions instead of probabilities, even though the docstring says this is ok. The underlying functional version `binary_precision_recall_curve` seems to work correctly in both cases. Both versions attempt to convert the logits to probabilities by passing them through a sigmoid *if any values are outside of the range [0, 1]*. In the object oriented case this condition is incorrectly checked independently for each batch, rather than for the metric as a whole. Consequently some batches may have sigmoid applied to their scores, while others do not, resulting in an incorrect curve for the dataset as a whole.
### To Reproduce
```py
import torch
import torchmetrics
b1 = torchmetrics.AUROC(task="binary")
b2 = torchmetrics.AUROC(task="binary")
score = torch.tensor([-1.0, 0.0, 1.0])
label = torch.tensor([1.0, 0.0, 1.0])
b1.update(score, label) # pass all three score/label pairs in as a single batch
for s, l in zip(score, label):
b2.update(torch.tensor([s]), torch.tensor([l])) # pass score/label pairs in one at a time
assert b1.compute().item() == 0.5
assert b2.compute().item() == 1.0
```
### Expected behavior
I would expect both AUCs to equal 0.5, as computed with scikit-learn using `sklearn.metrics.roc_auc_score(label, score)`.
### Environment
- TorchMetrics version (and how you installed TM, e.g. `conda`, `pip`, build from source): 1.3.0.post0
- Python & PyTorch Version (e.g., 1.0): 3.10.13
- Any other relevant information such as OS (e.g., Linux): MacOS
### Additional context
The bug is on line https://github.com/Lightning-AI/torchmetrics/blob/master/src/torchmetrics/classification/precision_recall_curve.py#L165 -- as it is currently written the function `_binary_precision_recall_curve_format` should only be called on the full dataset, not on individual batches. Otherwise the behavior is wrong if some batches have all scores in the range [0, 1] but other batches do not.
Some possible solutions are: (1) update the docs not to allow for logits in the object oriented interface, since the behavior is correct for probabilities; (2) don't try to automatically infer whether to apply sigmoid -- my choice, but would be a breaking change (3) refactor `_binary_precision_recall_curve_format` and accept that if any values are found which require sigmoid, then all values from past batches need to have sigmoid applied (this would be tricky in the case where thresholds are specified because the scores are not kept around).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read src/torchmetrics/classification/precision_recall_curve.py around line 165 and inspect _binary_precision_recall_curve_format, then run the supplied reproduction with batched and one-item updates. Done means the object-oriented and functional behavior is consistent for logits across batching, including the documented behavior and threshold-related cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100