huggingface / huggingface/candle
Quantization issue - Mixtral 8x22b
- Dominant language
- Rust
- Stars
- 21k
- Forks
- 1.8k
- Avg merge
- 16h 42m
- Merged PRs (30d)
- 25
Description
Hi all. I'm currently working to the implementation of a quantized version of Mixtral 8x22b. I'm using the weights from the following repo: [MaziyarPanahi/Mixtral-8x22B-Instruct-v0.1-GGUF](https://huggingface.co/MaziyarPanahi/Mixtral-8x22B-Instruct-v0.1-GGUF).
Unfortunately, expert-related tensors for each layer are merged (instead of having one for each expert). In order to use the available Mixtral implementation and exploit experts routing I:
- dequantize the qtensor;
- chunk the tensor obtained along dimension 0 and squeeze;
- quantize again to resume the conventional flow.
The problem arise during the quantization step. Considering `quantize` in `QTensor` implementation:
```rust
pub fn quantize(src: &Tensor, dtype: GgmlDType) -> Result {
let shape = src.shape();
let block_size = dtype.block_size();
check_shape(shape, block_size)?;
let src = src.to_dtype(crate::DType::F32)?.flatten_all()?;
let elem_count = shape.elem_count();
if elem_count % block_size != 0 {
crate::bail!(
"tensor size ({shape:?}) is not divisible by block size {}",
block_size
)
}
let mut storage = src.device().qzeros(elem_count, dtype)?;
storage.quantize(&src.storage())?;
Ok(Self {
storage,
shape: shape.clone(),
})
}
```
and `quantize` in `QStorage` implementation:
```rust
fn quantize(&mut self, src: &Storage) -> Result<()> {
match (self, src) {
(QStorage::Cpu(storage), Storage::Cpu(src)) => {
storage.from_float(src.as_slice::()?)?;
}
(QStorage::Metal(storage), Storage::Metal(src)) => storage.quantize(src)?,
(QStorage::Cuda(storage), Storage::Cuda(src)) => storage.quantize(src)?,
_ => crate::bail!("Invalid dequantize storage locations do not match"),
}
Ok(())
}
```
I found that the length of `src.as_slice::` is 8 times `elem_count`, causing problems with subsequent checks.
Any suggestions as to why this happens? Does the `chunk` operation have any influence?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.