Split converted model.bin into multiple .bin
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 4.7k
- Forks
- 536
- Avg merge
- 12h 12m
- Merged PRs (30d)
- 4
Description
Something that might be coming up more often with Models such as LLama-2-70b / Bloom etc. Is there any way to split the model.bin into multiple subfiles
Example: e.g in pytorch, the state dict is splitted.
model.bin = model-00001-of-00002.bin + model-00002-of-00002.bin
In my concrete case, even if llama-70b conversion is not yet possible due to their changed attention mechanisim - the conversion after int8 is ~65GB-70GB. At this large size, several services (cloudfare, huggingface) refuse to upload a model as a single file.
This would also enhance parallelization/download speed over network, e.g. in huggingface transformers also models < 50GB are split into sections of 9GB.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names no repository files, tests, or entry points. Start by reviewing the model conversion output and the PyTorch and Hugging Face splitting examples mentioned; done would mean defining and implementing a split model format that can be uploaded, downloaded, and loaded correctly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, huggingface, pytorch
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100