huggingface / huggingface/candle

Quantization issue - Mixtral 8x22b

Open
#2,201 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

Hi all. I'm currently working to the implementation of a quantized version of Mixtral 8x22b. I'm using the weights from the following repo: [MaziyarPanahi/Mixtral-8x22B-Instruct-v0.1-GGUF](https://huggingface.co/MaziyarPanahi/Mixtral-8x22B-Instruct-v0.1-GGUF).

Unfortunately, expert-related tensors for each layer are merged (instead of having one for each expert). In order to use the available Mixtral implementation and exploit experts routing I:
- dequantize the qtensor;
- chunk the tensor obtained along dimension 0 and squeeze;
- quantize again to resume the conventional flow.

The problem arise during the quantization step. Considering `quantize` in `QTensor` implementation:

```rust

pub fn quantize(src: &Tensor, dtype: GgmlDType) -> Result {
let shape = src.shape();
let block_size = dtype.block_size();
check_shape(shape, block_size)?;
let src = src.to_dtype(crate::DType::F32)?.flatten_all()?;
let elem_count = shape.elem_count();
if elem_count % block_size != 0 {
crate::bail!(
"tensor size ({shape:?}) is not divisible by block size {}",
block_size
)
}
let mut storage = src.device().qzeros(elem_count, dtype)?;
storage.quantize(&src.storage())?;
Ok(Self {
storage,
shape: shape.clone(),
})
}

```

and `quantize` in `QStorage` implementation:

```rust

fn quantize(&mut self, src: &Storage) -> Result<()> {
match (self, src) {
(QStorage::Cpu(storage), Storage::Cpu(src)) => {
storage.from_float(src.as_slice::()?)?;
}
(QStorage::Metal(storage), Storage::Metal(src)) => storage.quantize(src)?,
(QStorage::Cuda(storage), Storage::Cuda(src)) => storage.quantize(src)?,
_ => crate::bail!("Invalid dequantize storage locations do not match"),
}
Ok(())
}

```

I found that the length of `src.as_slice::` is 8 times `elem_count`, causing problems with subsequent checks.
Any suggestions as to why this happens? Does the `chunk` operation have any influence?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.