microsoft / microsoft/TransformerCompression

loading models is painful and not HF compatible

Open
#73 1 comment 0 reactions 1 assignee View on GitHub

@myshkov is already working on this.

Since Jan 23, 2024.

Dominant language
Python
Stars
460
Forks
58
Avg merge
8h 40m
Merged PRs (30d)
1

Description

To load a sliced model, we first load an uninitialized model, slice it, and load the checkpoint in. This is a pain for a few reasons:

  1. Adding new models means adding a switch in this code: https://github.com/microsoft/TransformerCompression/blob/a369325425cdfd3560c078060f829acd5ed8eb15/src/slicegpt/hf_utils.py#L75

  2. it's not easy for HF users to use our models directly, without running slicing themselves. It would be great if users could just do AutoModelForCausalLM.from_pretrained('microsoft/sliced-llama2-13B-30pc'). This would mean publishing such compatible models on HF, which would mean creating the model class explicitly.

Things to consider for a solution:

  • we'd probably need to store the "new hidden size" in the config somehow
  • we should make sure this doesn't block us from doing slicing with different levels per layer.
  • adding new models should "just work", without the current if/else in hf_utils
### Tasks
- [ ] https://github.com/microsoft/TransformerCompression/pull/102

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.