Lightning-AI / Lightning-AI/torchmetrics
torchmetrics.retrieval.RetrievalPrecisionRecallCurve plots precision calls it False Positive Rate
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 526
- Avg merge
- 6d 11h
- Merged PRs (30d)
- 5
Description
🐛 Bug
RetrievalPrecisionRecallCurve plots precision (x axis) and recall (y axis). However, the plot calls them False and True positive rates. While True positive rate is the same as recall, false positive rate is not the same as precision.
- why call it FPR/TPR instead of precision and recall which are the names used on the class?
- is the FPR label on the x axis wrong or is the computation of precision a mistake and it was meant actually to compute FPR?
As a minor issue, usually, recall is on the x axis and precision on the y axis (just google precision recall curves), and would be nice if they were swapped.
To Reproduce
Precision is true_positives / (true_positives + false _positives), recall is true_positives / real_positives, and FPR is false_positives / total_population`. Given this data:
pred = torch.tensor([1.0, .9, .8, .7, .6, .5, .4, .3, .2, .1])
target = torch.tensor([1, 1, 0, 1, 1, 0, 0, 1, 0, 0], dtype=int)
We should expect:
| Precision | Recall | FPR | |
|---|---|---|---|
| 0 | 1/1 | 1/5 | 0/10 |
| 1 | 2/2 | 2/5 | 0/10 |
| 2 | 2/3 | 2/5 | 1/10 |
| 3 | 3/4 | 3/5 | 1/10 |
| 4 | 4/5 | 4/5 | 1/10 |
| 5 | 4/6 | 4/5 | 2/10 |
| 6 | 4/7 | 4/5 | 3/10 |
| 7 | 5/8 | 5/5 | 3/10 |
| 8 | 5/9 | 5/5 | 4/10 |
| 9 | 5/10 | 5/5 | 5/10 |
yet, we get
>> metric = RetrievalPrecisionRecallCurve()
>> metric.update(pred, target,indexes=torch.zeros(pred.shape, dtype=int))
>> metric.compute()
(tensor([1.0000, 1.0000, 0.6667, 0.7500, 0.8000, 0.6667, 0.5714, 0.6250, 0.5556,
0.5000]),
tensor([0.2000, 0.4000, 0.4000, 0.6000, 0.8000, 0.8000, 0.8000, 1.0000, 1.0000,
1.0000]),
tensor([ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]))
>> fig, ax = metric.plot()
>> ax.set_xlim(0, 1)
>> ax.set_ylim(0, 1)
Environment
- TorchMetrics version 1.1.2
- Python 3.11.2
- OS: GNU/Linux Debian
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the provided example with RetrievalPrecisionRecallCurve and inspect its plot() output, including the axis labels and orientation. Trace how the computed precision and recall values are passed to the plot, then verify the result against the definitions in the issue. Done means the plot labels and axes accurately describe the returned values, with tests covering the example.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- data-visualization, machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100