Project-MONAI / Project-MONAI/MONAILabel
Need a scoring method to compare inference of 2 models
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 891
- Forks
- 269
- Avg merge
- 15h 41m
- Merged PRs (30d)
- 1
Description
Is your feature request related to a problem? Please describe.
This is related/ similar to https://github.com/Project-MONAI/MONAILabel/issues/1444
Describe the solution you'd like
After training a model_v1 then a new model_v2. User would want to compare where the models agreed and where it was improved.
Describe alternatives you've considered
Additional context
We need to consider multiple issues here as we do xor between the 2 inference and store a new inference with a new tag.
- For multiple labels we might need to loop over each labels
- need to count similar and difference pixel
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading related issue #1444 and tracing the existing inference storage and tagging flow. Clarify how model outputs should be compared across multiple labels, including agreement, differences, and pixel counts; done should mean the scoring behavior and stored comparison result are defined and covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100