Lightning-AI / Lightning-AI/torchmetrics

Interpolation bug for macro-averaged multiclass precision-recall curves (`torchmetrics.functional.classification.precision_recall_curve._multiclass_precision_recall_curve_compute`)

Open Beginner friendly
#3,413 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug / fix help wanted
Dominant language
Python
Stars
2.5k
Forks
526
Avg merge
6d 11h
Merged PRs (30d)
5

Description

## 🐛 Bug

When computing a macro-averaged precision-recall curve, [`_multiclass_precision_recall_curve`](https://github.com/Lightning-AI/torchmetrics/blob/master/src/torchmetrics/functional/classification/precision_recall_curve.py#L537) interpolates recall using precision values as `xp`, however [`interp`](https://github.com/Lightning-AI/torchmetrics/blob/master/src/torchmetrics/utilities/compute.py#L168) requires `xp` to be monotonically increasing, and precision is not guaranteed to be monotonic which can lead to weird-looking plots, especially when predictions are poorly correlated with the target.

**Micro-Average**
Image

**Macro-Average**
Image

#### Contrived Example
See the code, below. Here I deliberately generated data where precision doesn't vary monotonically to demonstrate how this leads to weird non-monotonic PR curves for `macro` averaging.

**Micro-Average**

Image

**Macro-Average**

Image

### Proposed Fix
The fix is very straightforward. We just need to swap round the interpolation (so we interpolate precision, using recall values rather than vice versa.) I'd be more than happy to submit a (draft) PR with an implementation if this would be helpful?

**Macro-Average After Fix (for first plot)**

Image

(_Note:_ Recall is now monotonic as is typical for a precision-recall curve)

**Macro-Average After Fix (for contrived example)**

Image

### To Reproduce

Sample code and steps to reproduce the behavior with expected result...

Code sample for Semi-Correlated Data

```python
import torch
import matplotlib.pyplot as plt
import torch.nn.functional as F
from torchmetrics.classification import MulticlassPrecisionRecallCurve

torch.manual_seed(42)

n_samples = 250
n_classes = 3
average = "micro" # Swap to view the 2 different curve-types

target = torch.randint(
low=0,
high=n_classes,
size=(n_samples,)
)

logits = torch.randn(n_samples, n_classes)

# Boost the true class so predictions are correlated with labels
boost = 0.1
logits[torch.arange(n_samples), target] += boost

preds = F.softmax(logits, dim=1)

pr_curve = MulticlassPrecisionRecallCurve(num_classes=n_classes, average=average)
pr_curve.update(preds, target)
curve = pr_curve.plot()
plt.show()
```

Code-Sample for Contrived Example

```python
import torch
import matplotlib.pyplot as plt
import torch.nn.functional as F
from torchmetrics.classification import MulticlassPrecisionRecallCurve

torch.manual_seed(42)

n_samples = 1000
n_classes = 3
average = "micro"

preds = torch.zeros(12, n_classes)
target = torch.zeros(12, dtype=torch.long)

# Block 1 - High confidence but wrong -> High FPs
for i in range(3):
preds[i] = torch.tensor([0.85 - i * 0.01, 0.10, 0.05])
target[i] = 1

# Block 2 - Medium confidence, correct -> Precision recovers
for i in range(3, 6):
preds[i] = torch.tensor([0.6 - i * 0.01, 0.25, 0.15])
target[i] = 0

# Block 3 - Lower confidence, incorrect -> Precision drops
for i in range(6, 9):
preds[i] = torch.tensor([[0.45 - (i - 6) * 0.01, 0.20, 0.35]])
target[i] = 2

# Block 4 - Lowest confidence, correct -> Precision recovers
for i in range(9, 12):
preds[i] = torch.tensor([[0.35 - (i - 9) * 0.01, 0.30, 0.35]])
target[i] = 2

pr_curve = MulticlassPrecisionRecallCurve(num_classes=n_classes, average=average)
pr_curve.update(preds, target)
curve = pr_curve.plot()
plt.show()

```

Environment

- TorchMetrics version (if build from source, add commit SHA): 1.9.0
- Python & PyTorch Version (e.g., 1.0): 3.14.5
- Any other relevant information such as OS (e.g., Linux): MacOS

### Additional context

N/A

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in src/torchmetrics/functional/classification/precision_recall_curve.py at _multiclass_precision_recall_curve_compute, then inspect interp in src/torchmetrics/utilities/compute.py. Reproduce the macro-average behavior with the provided examples and verify that the interpolation uses monotonic recall values and produces a typical monotonic-recall precision-recall curve.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.