deepspeedai / deepspeedai/DeepSpeed
[BUG] deepspeed_checkpoint.get_transformer_state has more PP_degree than real PP_degree
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
Describe the bug
when use deepspeed_checkpoint get transfromer_state, self.transformer_file_map has more pp_degree than real
To Reproduce
consider a transformer model with 70 transformer-layer, PP = 12
original code will set layers_per_pp as 70 // 12 = 5
now, transformer_file_map will have pp_degree as 70 // 5 = 14
Possible Fix
def _build_transformer_file_map(self):
transformer_layer_keys = self.layer_keys[1:-1]
file_map = {}
# XXX: this is not guaranteed
import math
layers_per_pp = math.ceil(len(transformer_layer_keys) / self.pp_degree)
if layers_per_pp == 0:
layers_per_pp = 1
...
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in deepspeed/checkpoint/deepspeed_checkpoint.py at _build_transformer_file_map and inspect how transformer_layer_keys and pp_degree determine the map. Reproduce the 70-layer, PP=12 case and verify that the resulting transformer_file_map does not contain more pipeline stages than requested.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100