Project-MONAI / Project-MONAI/MONAI

Implementing foundation models

Open
#7,771 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
8.7k
Forks
1.6k
Avg merge
5d 1h
Merged PRs (30d)
20

Description

Is your feature request related to a problem? Please describe.
I have been using MONAI for my project the last few months, the main problem is I need to use publicly available foundation models such as RAFT or MiDaS to generate intermidiate inputs for my models. This makes the implementation of such networks a bit tricky as you have to stick to certain input requirements (size, normalization, etc.) and implement those outside the MONAI bundles.

Describe the solution you'd like
Implementing a set of foundation models for MONAI. For my own application, I need depth estimation and optical flow estimation. However, I think it will benefitial to add models for segmentation and object detection also. I think there are two possible solutions:

  1. Create native foundation models in MONAI for these tasks.
  2. "Porting" some of the general foundation models available in Pytorch such as RAFT or MiDaS and implement them inside the MONAI framework.

Additional context
I'd like to use this post as a discussion about the implementation of such foundation models. I am more than happy to participate on the process, as it would massively benefit my own work.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names no files, tests, or entry points and presents foundation-model support as an open discussion. First narrow the scope to specific models and tasks, then define the integration point and acceptance criteria before implementation can be considered complete.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.