Project-MONAI / Project-MONAI/tutorials

maisi unconditioned inference return noise

Open
#2,017 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Jupyter Notebook
Stars
2.5k
Forks
803
Avg merge
6d 22h
Merged PRs (30d)
3

Description

Hallo,

I tried longer to make the diffusion unet (rectified flow version) work but without success. the VAE performes correct and encoding and reconstruction look fine. I take all the configs from the repo so configuration as well as weights are equivalently loaded as in https://github.com/Project-MONAI/tutorials/blob/main/generation/maisi/maisi_inference_tutorial.ipynb.

Using the diffusion unet with the existing code of diffusion_model_infer.py with

from scripts.diff_model_infer import run_inference
data = run_inference(
  args,
  device,
  autoencoder,
  diffusion_unet,
  1.4,#scale_factor,
  None,#top_region_index_tensor,
  None, #bottom_region_index_tensor,
  torch.tensor([[1.5,1.5,4.0]]).to('cuda'),#spacing_tensor,
  torch.tensor([1]).to('cuda'),#modality_tensor,
  (96,96,96),#output_size,
  4,
  logger,
)

I saw that there was an issue with the architecture which was different during training and the implementation but I'm not sure if that is related.
https://github.com/Project-MONAI/MONAI/issues/7991.
Some observations I made is when I would like to just reconstruction an image and basically make one step, the recon get's extremly noise.

For any suggestion what could be the potential issue, I'm thankful.

Best,
Lukas

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with scripts/diff_model_infer.py and the MAISI inference tutorial notebook, reproducing the one-step reconstruction that produces noise. Compare the inference architecture and loaded configuration with the training implementation, including the differences discussed in issue 7991. Done means the unconditioned inference or reconstruction no longer returns noisy output under the supplied settings.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook, python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.