Project-MONAI / Project-MONAI/tutorials
Training time of MAISI VAE
Nobody has claimed this yet.
- Dominant language
- Jupyter Notebook
- Stars
- 2.5k
- Forks
- 803
- Avg merge
- 6d 22h
- Merged PRs (30d)
- 3
Description
Hi, I am trying to train the MAISI VAE from scratch using my dataset.
I wonder how long it takes to train on such a large dataset?
When I use 30k CT scans, it takes nearly 20 hours for 1 epoch on two A100. Is it correct? I use DDP, but I found it was even slower when I increased the GPU number.
If so, is there any method to accelerate the training process?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the MAISI VAE training entry point and its DDP setup, then reproduce the reported 20-hour epoch on 30k CT scans using two A100 GPUs. No file or test is identified; done would require establishing whether the timing and scaling are expected and documenting a supported acceleration approach.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100