deepspeedai / deepspeedai/DeepSpeedExamples
Load model error in step3
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 6.8k
- Forks
- 1.1k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 1
Description
When setting zero_stage=3, load my own ckpt in .pt format:
model_config = AutoConfig.from_pretrained(model_name_or_path)
model = AutoModel.from_config(model_config)
the shape of param in model is all torch.Size([0]), because the error message is:
size mismatch for model.decoder.layers.1.self_attn.k_proj.weight: copying a param with shape torch.Size([5120, 5120]) from checkpoint, the shape in current model is torch.Size([0]).
The ckpt I load is with correct shape of params, but the model build from config seems wrong.
With other zero_stages, the error will not occur.
I run the script line by line, the error does not occur either.
Anyone come across this problem?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce loading the .pt checkpoint with zero_stage=3, starting from the AutoConfig.from_pretrained and AutoModel.from_config calls shown in the issue. Compare parameter shapes with other zero stages and with line-by-line execution; done means explaining why parameters become torch.Size([0]) and establishing correct checkpoint loading behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100