Project-MONAI / Project-MONAI/MetricsReloaded
Assignment strategy (FP2.5.8) - double assignment, one prediction for multiple references.. also covered?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 104
- Forks
- 18
- PR merge metrics
- No merged PRs in 30d
Description
Dear Metric Reloaded Team,
I have worked with Brain metastases data, and I was confused when I evaluated lesion-wise (instance-level) detection metrics (F1 -score) from predicted segmentation. Currently, I match each lesion if there is overlap, or overlap(DSC) > 0.1. And I observed that there are quite multiple assigned cases in both ways (one pred - multiple GTs, multiple pred - one GT). That's how I visit your great work (Metric Reloaded) to get some guidance.
I have briefly checked resolve ambiguity code, but I'm still not sure whether this code also treat the remained GTs as FN or ignore them. Or counting TPs is always based on GTs?
I illustrated this issue with the image in the below repository as well, since I used their code.
https://github.com/rachitsaluja/BraTS-2023-Metrics/issues/11
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with MetricsReloaded/utility/assignment_localization.py around the referenced resolve ambiguity code at line 510, then compare the question with the linked BraTS-2023-Metrics issue. Determine the intended treatment of unmatched references and predictions and how TP, FP, and FN counts are defined; done requires a clear documented or tested resolution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- computer-vision, machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100