modelscope / modelscope/DiffSynth-Studio
LoRA training of 14B model is slow~(40s/it). Observation: The forward pass time varies significantly across different DiT blocks.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 13.1k
- Forks
- 1.3k
- Avg merge
- 13h 12m
- Merged PRs (30d)
- 45
Description
Hi, I have tried I2V-14b LoRA training, with gradient_checkpointing=True, use_gradient_checkpointing_offload=False. I noticed that the training speed is approximately 40 seconds per iteration. Is this normal 🤔? (1xA800, without deepspeed)
I further visualize time cost for 1. vae encode, 2. forward and 3. backward stages. I find forward stage will cost about 9 second (and backward is about 2~3 times). Specifically, for total 40 blocks, the first 10 blocks will cost 0.00x second, but the later 30 blocks will cost 0.2 second
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No file, test, or entry point is named. Start by reproducing I2V-14b LoRA training with the stated gradient-checkpointing settings and profiling VAE encode, forward, backward, and individual DiT blocks; done means determining whether the timing is expected and documenting or addressing the cause.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100