Lightning-AI / Lightning-AI/torchmetrics
Add Precision-Recall-Gain curve, Area Under Precision Recall Gain curve, and FGain1 score
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 526
- Avg merge
- 6d 11h
- Merged PRs (30d)
- 5
Description
## 🚀 Feature
Add Precision-Recall-Gain (PRG) curve as a new feature with the same interface as the Precision-Recall (PR) curve.
Along with PRG, the Area Under the Precision Recall Gain curve (AUPRG) can be calculated, like is done `AveragePrecision`.
The FGain1 score (FG1) is the F1 score, but transformed such that it is the minor diagonal in PRG-space. This could be added.
### Motivation
The PR curve has some caveats as described in [1]. PRG aims to fix these problems:
1. baselines are non-universal
2. interpolation is non-linear
3. F-isometrics are non-linear
4. Pareto-front is non-convex
5. Area under PR curve does not relate to the expected F + there is an unachievable region
In particular, the area under the PR curve is demonstrated to sometimes favour models that result in lower F1-scores. The PRG curve will ultimately result in better model selection.
### Pitch
A Torchmetrics implementation of the PRG curve that has the same interface as the PR curve would aid in better model selection.
```python
>>> pred = torch.tensor([0, 0.1, 0.8, 0.4])
>>> target = torch.tensor([0, 1, 1, 0])
>>> prg_curve = PrecisionRecallGainCurve(task="binary")
>>> precision_gain, recall_gain, thresholds = prg_curve(pred, target)
>>> precision_gain
tensor([1.0000, 0.0000, 0.5000, 0.0000])
>>> recall_gain
tensor([0.0000, 0.0000, 1.0000, 1.0000])
>>> thresholds
...
```
Precision-Gain (PG) and Recall-Gain (RG) can be calculated as
$$
PG = 1 - \frac{tp + fn}{fp + tn} \cdot \frac{fp}{tp},
$$
and
$$
RG = 1 - \frac{tp + fn}{fp + tn} \cdot \frac{fn}{tp}.
$$
AUPRG can be calculated as done with `AveragePrecision`, but only accounting for the area in PR & RG $\in [0, 1]$.
FG1 can be calculated as
$$
FG_1 = \frac{1}{2} PG + \frac{1}{2} RG.
$$
It would be even more awesome if PRG can be extended to the multiclass/multilabel case.
### Alternatives
The original authors of [1] have developed a package, [pyprg](https://github.com/meeliskull/prg) (which is out-of-date with dependencies).
```
pip instal pyprg
```
Then,
```python
from prg import prg
prg_curve = prg.create_prg_curve(labels=targets, scores=prediction)
precision_gain = prg_curve["precision_gain"]
recall_gain = prg_curve["recall_gain"]
auprg= prg.calc_auprg(prg_curve)
```
### Additional context
[1] Flach & Kull. http://people.cs.bris.ac.uk/~flach/PRGcurves/PRcurves.pdf
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the existing Precision-Recall curve and AveragePrecision implementations in Torchmetrics, then compare their interfaces with the proposed PrecisionRecallGainCurve example. Implementing the binary PRG curve, AUPRG, and FGain1 would be complete when the supplied formulas and example outputs are supported; multiclass and multilabel extension is optional context.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100