the shape of params lm_head.wegiht is not compatible between base weight and delta weight
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.5k
- Forks
- 4.8k
- PR merge metrics
- No merged PRs in 30d
Description
when running apply_delta.py of weight version v1.1, I find that the the shape of params lm_head.wegiht is [32000, 5120] in base weight, and is [32001, 5120] in delta weight. However, the shape of model.embed_tokends.weight are the same for these two weights.
Have anyone met this problem before?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with apply_delta.py and reproduce the v1.1 run using the reported base and delta weights. Inspect how lm_head.wegiht and model.embed_tokends.weight shapes are loaded and compared; done means identifying the cause of the [32000, 5120] versus [32001, 5120] mismatch and documenting or fixing the behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100