Lightning-AI / Lightning-AI/torchmetrics

Add Precision-Recall-Gain curve, Area Under Precision Recall Gain curve, and FGain1 score

Open
#1,775 6 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
2.5k
Forks
526
Avg merge
6d 11h
Merged PRs (30d)
5

Description

## 🚀 Feature

Add Precision-Recall-Gain (PRG) curve as a new feature with the same interface as the Precision-Recall (PR) curve.

Along with PRG, the Area Under the Precision Recall Gain curve (AUPRG) can be calculated, like is done `AveragePrecision`.

The FGain1 score (FG1) is the F1 score, but transformed such that it is the minor diagonal in PRG-space. This could be added.

### Motivation

The PR curve has some caveats as described in [1]. PRG aims to fix these problems:
1. baselines are non-universal
2. interpolation is non-linear
3. F-isometrics are non-linear
4. Pareto-front is non-convex
5. Area under PR curve does not relate to the expected F + there is an unachievable region

In particular, the area under the PR curve is demonstrated to sometimes favour models that result in lower F1-scores. The PRG curve will ultimately result in better model selection.

### Pitch

A Torchmetrics implementation of the PRG curve that has the same interface as the PR curve would aid in better model selection.

```python
>>> pred = torch.tensor([0, 0.1, 0.8, 0.4])
>>> target = torch.tensor([0, 1, 1, 0])
>>> prg_curve = PrecisionRecallGainCurve(task="binary")
>>> precision_gain, recall_gain, thresholds = prg_curve(pred, target)
>>> precision_gain
tensor([1.0000, 0.0000, 0.5000, 0.0000])
>>> recall_gain
tensor([0.0000, 0.0000, 1.0000, 1.0000])
>>> thresholds
...
```

Precision-Gain (PG) and Recall-Gain (RG) can be calculated as

$$
PG = 1 - \frac{tp + fn}{fp + tn} \cdot \frac{fp}{tp},
$$

and

$$
RG = 1 - \frac{tp + fn}{fp + tn} \cdot \frac{fn}{tp}.
$$

AUPRG can be calculated as done with `AveragePrecision`, but only accounting for the area in PR & RG $\in [0, 1]$.

FG1 can be calculated as

$$
FG_1 = \frac{1}{2} PG + \frac{1}{2} RG.
$$

It would be even more awesome if PRG can be extended to the multiclass/multilabel case.

### Alternatives

The original authors of [1] have developed a package, [pyprg](https://github.com/meeliskull/prg) (which is out-of-date with dependencies).
```
pip instal pyprg
```
Then,
```python
from prg import prg
prg_curve = prg.create_prg_curve(labels=targets, scores=prediction)
precision_gain = prg_curve["precision_gain"]
recall_gain = prg_curve["recall_gain"]
auprg= prg.calc_auprg(prg_curve)
```

### Additional context

[1] Flach & Kull. http://people.cs.bris.ac.uk/~flach/PRGcurves/PRcurves.pdf

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the existing Precision-Recall curve and AveragePrecision implementations in Torchmetrics, then compare their interfaces with the proposed PrecisionRecallGainCurve example. Implementing the binary PRG curve, AUPRG, and FGain1 would be complete when the supplied formulas and example outputs are supported; multiclass and multilabel extension is optional context.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.