huggingface / huggingface/candle
[QUESTION] Initialize VarBuilder from binary data from multiple safetensor files
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 21.1k
- Forks
- 1.8k
- Avg merge
- 16h 42m
- Merged PRs (30d)
- 25
Description
Hi, I am playing around with the llama example using the Llama-2-7b-hf model.
In the candle source code, it initializes VarBuilder from files by `let vb = unsafe { VarBuilder::from_mmaped_safetensors(&filenames, dtype, &device)? };`
I have all the necessary files read and stored as bytes, so I'm looking into using the function `from_buffered_safetensors(data: Vec, dtype: DType, dev: &Device)` in `var_builder.rs`. However, I'm not sure how to pass in the bytes of more than one file into this. I tried to combine the bytes of two safetensors file into one by simply extending the two `Vec` but it is not working.
Any help would be appreciated!
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read var_builder.rs and compare from_mmaped_safetensors with from_buffered_safetensors, focusing on how multiple safetensor files are represented. Reproduce the Llama-2-7b-hf loading scenario with in-memory bytes; done means establishing and documenting the supported approach for multiple files.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100