Lightning-AI / Lightning-AI/torchmetrics
Interpolation bug for macro-averaged multiclass precision-recall curves (`torchmetrics.functional.classification.precision_recall_curve._multiclass_precision_recall_curve_compute`)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 526
- Avg merge
- 6d 11h
- Merged PRs (30d)
- 5
Description
## 🐛 Bug
When computing a macro-averaged precision-recall curve, [`_multiclass_precision_recall_curve`](https://github.com/Lightning-AI/torchmetrics/blob/master/src/torchmetrics/functional/classification/precision_recall_curve.py#L537) interpolates recall using precision values as `xp`, however [`interp`](https://github.com/Lightning-AI/torchmetrics/blob/master/src/torchmetrics/utilities/compute.py#L168) requires `xp` to be monotonically increasing, and precision is not guaranteed to be monotonic which can lead to weird-looking plots, especially when predictions are poorly correlated with the target.
**Micro-Average**
**Macro-Average**
#### Contrived Example
See the code, below. Here I deliberately generated data where precision doesn't vary monotonically to demonstrate how this leads to weird non-monotonic PR curves for `macro` averaging.
**Micro-Average**
**Macro-Average**
### Proposed Fix
The fix is very straightforward. We just need to swap round the interpolation (so we interpolate precision, using recall values rather than vice versa.) I'd be more than happy to submit a (draft) PR with an implementation if this would be helpful?
**Macro-Average After Fix (for first plot)**
(_Note:_ Recall is now monotonic as is typical for a precision-recall curve)
**Macro-Average After Fix (for contrived example)**
### To Reproduce
Sample code and steps to reproduce the behavior with expected result...
Code sample for Semi-Correlated Data
```python
import torch
import matplotlib.pyplot as plt
import torch.nn.functional as F
from torchmetrics.classification import MulticlassPrecisionRecallCurve
torch.manual_seed(42)
n_samples = 250
n_classes = 3
average = "micro" # Swap to view the 2 different curve-types
target = torch.randint(
low=0,
high=n_classes,
size=(n_samples,)
)
logits = torch.randn(n_samples, n_classes)
# Boost the true class so predictions are correlated with labels
boost = 0.1
logits[torch.arange(n_samples), target] += boost
preds = F.softmax(logits, dim=1)
pr_curve = MulticlassPrecisionRecallCurve(num_classes=n_classes, average=average)
pr_curve.update(preds, target)
curve = pr_curve.plot()
plt.show()
```
Code-Sample for Contrived Example
```python
import torch
import matplotlib.pyplot as plt
import torch.nn.functional as F
from torchmetrics.classification import MulticlassPrecisionRecallCurve
torch.manual_seed(42)
n_samples = 1000
n_classes = 3
average = "micro"
preds = torch.zeros(12, n_classes)
target = torch.zeros(12, dtype=torch.long)
# Block 1 - High confidence but wrong -> High FPs
for i in range(3):
preds[i] = torch.tensor([0.85 - i * 0.01, 0.10, 0.05])
target[i] = 1
# Block 2 - Medium confidence, correct -> Precision recovers
for i in range(3, 6):
preds[i] = torch.tensor([0.6 - i * 0.01, 0.25, 0.15])
target[i] = 0
# Block 3 - Lower confidence, incorrect -> Precision drops
for i in range(6, 9):
preds[i] = torch.tensor([[0.45 - (i - 6) * 0.01, 0.20, 0.35]])
target[i] = 2
# Block 4 - Lowest confidence, correct -> Precision recovers
for i in range(9, 12):
preds[i] = torch.tensor([[0.35 - (i - 9) * 0.01, 0.30, 0.35]])
target[i] = 2
pr_curve = MulticlassPrecisionRecallCurve(num_classes=n_classes, average=average)
pr_curve.update(preds, target)
curve = pr_curve.plot()
plt.show()
```
Environment
- TorchMetrics version (if build from source, add commit SHA): 1.9.0
- Python & PyTorch Version (e.g., 1.0): 3.14.5
- Any other relevant information such as OS (e.g., Linux): MacOS
### Additional context
N/A
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in src/torchmetrics/functional/classification/precision_recall_curve.py at _multiclass_precision_recall_curve_compute, then inspect interp in src/torchmetrics/utilities/compute.py. Reproduce the macro-average behavior with the provided examples and verify that the interpolation uses monotonic recall values and produces a typical monotonic-recall precision-recall curve.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100