Lightning-AI / Lightning-AI/torchmetrics
Add a new metric: Localization Recall Precision (LRP)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 526
- Avg merge
- 6d 11h
- Merged PRs (30d)
- 5
Description
## 🚀 Feature
Add a new metric: Localisation Recall Precision (LRP)
This metric provides a unified evaluation of object detection performance by jointly considering localisation error, recall, and precision. It is particularly useful as an alternative to mAP for assessing object detectors.
### Motivation
In my current and past projects, I have relied heavily on torchmetrics for evaluating deep learning models, especially for object detection tasks. While the library provides common metrics such as mAP, I also use Localisation Recall Precision (LRP), which provides a more interpretable and fine-grained assessment of detection quality.
LRP measures performance across false positives, false negatives, and localisation error simultaneously. I think it's a nice metric and including it in torchmetrics would make it more accessible for users working with object detection models.
### Pitch
**I would like to contribute by implementing Localisation Recall Precision** as a new metric within torchmetrics.
The initial implementation would:
Support standard object detection evaluation input formats (predictions and targets with bounding boxes, scores, and labels).
Compute LRP error.
Follow the existing structure and design of object detection metrics in the repo (similar to mAP).
**I would be happy to open a PR with an initial implementation and tests once the proposal is approved.**
### Alternatives
You could compute LRP outside of torchmetrics using standalone scripts. But this is less cool.
Relying solely on mAP.
### Additional context
The metric was first introduced in “Localisation Recall Precision (LRP): A New Performance Metric for Object Detection” (https://arxiv.org/pdf/1807.01696), and has since been used as an alternative evaluation metric in several object detection benchmarks and research papers.
I believe including this would expand the usefulness of torchmetrics for users working on detection tasks.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Review the existing object detection metrics, especially mAP, and read the cited LRP paper first. Define the supported prediction and target inputs, the expected LRP behavior, and the tests needed to show the new metric is complete.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100