microsoft / microsoft/TransformerCompression
loading models is painful and not HF compatible
@myshkov is already working on this.
Since Jan 23, 2024.
- Dominant language
- Python
- Stars
- 460
- Forks
- 58
- Avg merge
- 8h 40m
- Merged PRs (30d)
- 1
Description
To load a sliced model, we first load an uninitialized model, slice it, and load the checkpoint in. This is a pain for a few reasons:
-
Adding new models means adding a switch in this code: https://github.com/microsoft/TransformerCompression/blob/a369325425cdfd3560c078060f829acd5ed8eb15/src/slicegpt/hf_utils.py#L75
-
it's not easy for HF users to use our models directly, without running slicing themselves. It would be great if users could just do
AutoModelForCausalLM.from_pretrained('microsoft/sliced-llama2-13B-30pc'). This would mean publishing such compatible models on HF, which would mean creating the model class explicitly.
Things to consider for a solution:
- we'd probably need to store the "new hidden size" in the config somehow
- we should make sure this doesn't block us from doing slicing with different levels per layer.
- adding new models should "just work", without the current if/else in hf_utils
### Tasks
- [ ] https://github.com/microsoft/TransformerCompression/pull/102
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.