Project-MONAI / Project-MONAI/MONAI

Create a confusion matrix based on tumors not per pixel for segmentation models

Open
#8,197 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
8.7k
Forks
1.6k
Avg merge
5d 1h
Merged PRs (30d)
20

Description

Is your feature request related to a problem? Please describe.
Confusion matrix as true positive (TP), false positive (FP), true negative (TN) and false negative (FN) is a powerful accuracy metric that we have for classification.
We also have it segmentation tasks based on pixel counts. However, this is not meaningful for radiologist or data scientist since they care about number of tumors missed not number of pixel missed.

Describe the solution you'd like
It would be great if we have an accuracy metric that would do: for each label calculate connected component and compare it with the ground truth. If there is more than x % overlap then it is calculated as TP. if there is no ground truth then it is FP. Then we should check on the ground truth that doesn't have any inference should be counted as FN
We can ignore TN as it is about the background.

Describe alternatives you've considered
Manually have radiologist count the tumors and write values in excel files. This doesn't scale at all

Additional context

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating MONAI's existing segmentation confusion-matrix metrics and any connected-component utilities. Read how current metrics define true and false positives, then determine how tumor-level overlap and unmatched ground-truth components should be represented. Done means a documented metric counts TP, FP, and FN per tumor while excluding TN.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
computer-vision, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.