Project-MONAI / Project-MONAI/MONAI

Enable Volume Generation from Concrete Labelmaps in MAISI ControlNet

Open
#8,314 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
8.7k
Forks
1.6k
Avg merge
5d 1h
Merged PRs (30d)
20

Description

Is your feature request related to a problem? Please describe.
Image generation with MAISI

Describe the solution you'd like
Afaiu, currently, MAISI can "only" generate volumes conditionally on controlnet_cond. These are relative size changes of certain anatomical labels. But it cannot generate a volume from a concrete labelmap that is given.

Describe alternatives you've considered
In ControlNet's forward() method, I did not see an alternative that would allow embedding a concrete labelmap. The Diffusion Model requires embeddings from ControlNet, and cannot accept concrete labelmaps either.

Additional context
Figure 2 in the MAISI paper also seems to indicate that ControlNet can only accept "task-specific conditions", and "Diffusion Model" depends on ControlNet embeddings.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading monai/apps/generation/maisi/networks/controlnet_maisi.py, especially ControlNet's forward() method, and compare it with the MAISI paper's description of task-specific conditions. Determine how a concrete labelmap would be represented and passed through ControlNet and the diffusion model; done means volume generation accepts the supplied labelmap as a condition and the behavior is covered by appropriate tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
ai, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.