huggingface / huggingface/candle

Quantized tensors load support with candle_nn::VarBuilder

Open
#1,877 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

there is multiple data type tensors in the quantized models(fp16, int32..), but `candle_nn::VarBuilder` only use same dtype to load all tensors. test with [llama awq](https://huggingface.co/TheBloke/Llama-2-13B-chat-AWQ)

eg:
```rust
#[test]
fn test_varbb() -> candle_core::Result<()> {
let device = candle_core::Device::new_cuda(0).unwrap();

let model_weight_files = vec!["./model.safetensors"];
let vb = unsafe {
candle_nn::VarBuilder::from_mmaped_safetensors(&model_weight_files, DType::F16, &device)?
};
let test_tensor0 = vb.pp("model.layers.1.self_attn.q_proj");
let test_tensor0 = test_tensor0.get(((5120_usize, 640_usize)), "qweight")?; // failed to get tensor with wrong dtype
println!("{:?}", test_tensor0.dtype());

let test_tensor1 = vb.pp("model.layers.0.input_layernorm");
let test_tensor1 = test_tensor1.get(5120_usize, "weight")?;
println!("{:?}", test_tensor1.dtype());

Ok(())
}
```

with python
```python
#!/usr/bin/python3

from safetensors.numpy import save_file, load_file

loaded = load_file("./model.safetensors")
x=loaded['model.layers.1.self_attn.q_proj.qweight']
print(x.dtype, x.shape)

y=loaded['model.layers.0.input_layernorm.weight']
print(y.dtype, y.shape)
```

```
int32 (5120, 640)
float16 (5120,)
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with candle_nn::VarBuilder::from_mmaped_safetensors and reproduce the provided test using the Llama AWQ model. Check how qweight and input_layernorm.weight are loaded when their dtypes differ; done means both tensors load successfully with their stored dtypes.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.