modelscope / modelscope/DiffSynth-Studio
使用deepspeed zero_stage 3训练，遇到张量shape不一致的问题该怎么解决？
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 13.1k
- Forks
- 1.3k
- Avg merge
- 13h 12m
- Merged PRs (30d)
- 45
Description
同样的工程使用deepspeed zero_stage 2训练，不会出现shape不一致的问题，但会出现显存OOM。
推测是zero_stage 3分片后导致张量shape改变，求助解决方案
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the same project with DeepSpeed ZeRO stage 2 and stage 3, then capture the operation and tensor shapes reported by the mismatch. Compare the memory behavior as well; done means training no longer raises a shape-mismatch error while addressing the stage 2 OOM.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100