huggingface / huggingface/candle

Unable to load Quantized mistral

Open
#2,427 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

I am able to load quantised_mistral

For the model_id and revision I have chosen this
```
let model_id = "mistralai/Mistral-7B-v0.1".to_string();
let revision = "26bca36bde8333b5d7f72e9ed20ccda6a618af24".to_string();
```

let filenames = hub_load_safetensors(&api_repo, safetensors_file_name)?;
where filenames is of Vec

What should I choose for the Varbuilder?

When I choose the Varbuilder form the quantised_mistral file?

QuantizedMistralVarBuilder::from_gguf(&filenames, &device);

I am getting this error

```
error[E0277]: the trait bound `Vec: AsRef` is not satisfied
--> src/model/backends/candle/text_generation.rs:39:57
|
39 | QuantizedMistralVarBuilder::from_gguf(&filenames, &device);
| ------------------------------------- ^^^^^^^^^ the trait `AsRef` is not implemented for `Vec`, which is required by `&Vec: AsRef`
| |
| required by a bound introduced by this call
|
= help: the following other types implement trait `AsRef`:
as AsRef>>
as AsRef<[T]>>
= note: required for `&Vec` to implement `AsRef`
```

And If I use

`candle_nn::VarBuilder;
`
let vb = unsafe { VarBuilder::from_mmaped_safetensors(&filenames, dtype, &device)? };
QuantizedMistral::new(&config, vb)?

I get
```

error[E0308]: mismatched types
--> src/model/backends/candle/text_generation.rs:41:44
|
41 | QuantizedMistral::new(&config, vb)?
| --------------------- ^^ expected `VarBuilder`, found `VarBuilderArgs<'_, Box<...>>`
| |
| arguments to this function are incorrect
|
= note: expected struct `VarBuilder`
found struct `VarBuilderArgs<'_, Box>`
```

What am I doing wrong here?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in src/model/backends/candle/text_generation.rs and inspect the signatures and expected types for QuantizedMistralVarBuilder::from_gguf, VarBuilder::from_mmaped_safetensors, and QuantizedMistral::new. Compare those APIs with the Vec and VarBuilder values shown in the errors. Done means the selected builder compiles and can load the specified quantized Mistral model.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.