deepspeedai / deepspeedai/DeepSpeed
[REQUEST] Does Universal Checkpoint supports for MoE Checkpoint？
Open
@tohtana is already working on this.
Since Jul 2, 2024.
enhancement
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
Is your feature request related to a problem? Please describe.
When I want to convert my MoE checkpoint with ZeRO 1 to Universal Checkpoint, I meet a error named 'KeyError: 'param_shapes''. So I am curious that does Universal Checkpoint supports for MoE Checkpoint?
Describe the solution you'd like
I hope I could use Universal Checkpoint to resume training for MoE checkpoint with different GPUs. THX.
Describe alternatives you've considered
Another way to resume training for MoE checkpoint with different GPUs.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.