bigscience-workshop / bigscience-workshop/Megatron-DeepSpeed
About convert deepspeed to deepspeed checkpoint
Open
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 226
- PR merge metrics
- No merged PRs in 30d
Description
In training, I used swiglu, TP=4, PP=2. I use deepspeed_to_deepspeed.py to convert the checkpoint to a TP=1, PP=1 one. When evaluating the obtained checkpoint, it is found that the accuracy is inconsistent with the previous one.
（When I do not use swiglu, I can get the correct checkpoint）
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.