huggingface / huggingface/candle
Quantized tensors load support with candle_nn::VarBuilder
- Dominant language
- Rust
- Stars
- 21k
- Forks
- 1.8k
- Avg merge
- 16h 42m
- Merged PRs (30d)
- 25
Description
there is multiple data type tensors in the quantized models(fp16, int32..), but `candle_nn::VarBuilder` only use same dtype to load all tensors. test with [llama awq](https://huggingface.co/TheBloke/Llama-2-13B-chat-AWQ)
eg:
```rust
#[test]
fn test_varbb() -> candle_core::Result<()> {
let device = candle_core::Device::new_cuda(0).unwrap();
let model_weight_files = vec!["./model.safetensors"];
let vb = unsafe {
candle_nn::VarBuilder::from_mmaped_safetensors(&model_weight_files, DType::F16, &device)?
};
let test_tensor0 = vb.pp("model.layers.1.self_attn.q_proj");
let test_tensor0 = test_tensor0.get(((5120_usize, 640_usize)), "qweight")?; // failed to get tensor with wrong dtype
println!("{:?}", test_tensor0.dtype());
let test_tensor1 = vb.pp("model.layers.0.input_layernorm");
let test_tensor1 = test_tensor1.get(5120_usize, "weight")?;
println!("{:?}", test_tensor1.dtype());
Ok(())
}
```
with python
```python
#!/usr/bin/python3
from safetensors.numpy import save_file, load_file
loaded = load_file("./model.safetensors")
x=loaded['model.layers.1.self_attn.q_proj.qweight']
print(x.dtype, x.shape)
y=loaded['model.layers.0.input_layernorm.weight']
print(y.dtype, y.shape)
```
```
int32 (5120, 640)
float16 (5120,)
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with candle_nn::VarBuilder::from_mmaped_safetensors and reproduce the provided test using the Llama AWQ model. Check how qweight and input_layernorm.weight are loaded when their dtypes differ; done means both tensors load successfully with their stored dtypes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100