Lightning-AI / Lightning-AI/torchmetrics

torchmetrics.retrieval.RetrievalPrecisionRecallCurve plots precision calls it False Positive Rate

Open
#3,403 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug / fix help wanted
Dominant language
Python
Stars
2.5k
Forks
526
Avg merge
6d 11h
Merged PRs (30d)
5

Description

🐛 Bug

RetrievalPrecisionRecallCurve plots precision (x axis) and recall (y axis). However, the plot calls them False and True positive rates. While True positive rate is the same as recall, false positive rate is not the same as precision.

  1. why call it FPR/TPR instead of precision and recall which are the names used on the class?
  2. is the FPR label on the x axis wrong or is the computation of precision a mistake and it was meant actually to compute FPR?

As a minor issue, usually, recall is on the x axis and precision on the y axis (just google precision recall curves), and would be nice if they were swapped.

To Reproduce

Precision is true_positives / (true_positives + false _positives), recall is true_positives / real_positives, and FPR is false_positives / total_population`. Given this data:

pred = torch.tensor([1.0, .9, .8, .7, .6, .5, .4, .3, .2, .1])
target = torch.tensor([1, 1, 0, 1, 1, 0, 0, 1, 0, 0], dtype=int)

We should expect:

Precision Recall FPR
0 1/1 1/5 0/10
1 2/2 2/5 0/10
2 2/3 2/5 1/10
3 3/4 3/5 1/10
4 4/5 4/5 1/10
5 4/6 4/5 2/10
6 4/7 4/5 3/10
7 5/8 5/5 3/10
8 5/9 5/5 4/10
9 5/10 5/5 5/10

yet, we get

>> metric = RetrievalPrecisionRecallCurve()
>> metric.update(pred, target,indexes=torch.zeros(pred.shape, dtype=int))
>> metric.compute()
(tensor([1.0000, 1.0000, 0.6667, 0.7500, 0.8000, 0.6667, 0.5714, 0.6250, 0.5556,
         0.5000]),
 tensor([0.2000, 0.4000, 0.4000, 0.6000, 0.8000, 0.8000, 0.8000, 1.0000, 1.0000,
         1.0000]),
 tensor([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10]))
>> fig, ax = metric.plot()
>> ax.set_xlim(0, 1)
>> ax.set_ylim(0, 1)
Image
Environment
  • TorchMetrics version 1.1.2
  • Python 3.11.2
  • OS: GNU/Linux Debian

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the provided example with RetrievalPrecisionRecallCurve and inspect its plot() output, including the axis labels and orientation. Trace how the computed precision and recall values are passed to the plot, then verify the result against the definitions in the issue. Done means the plot labels and axes accurately describe the returned values, with tests covering the example.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
data-visualization, machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.