[Feature] Will multi-modal models support W8A8 quantization in the future?
Open
- Dominant language
- Python
- Stars
- 8.1k
- Forks
- 748
- Avg merge
- 6d 2h
- Merged PRs (30d)
- 54
Description
### Motivation
Our business model (Internvl 2-26B) outputs very few tokens (1-2 tokens) after prompt optimization, which can be considered as only the prefill stage. Therefore, we hope to use W8A8 quantization to speed up the inference process. However, we found that lmdeploy does not support W8A8 inference for multi-modal models: [https://github.com/InternLM/lmdeploy/issues/2042.](https://github.com/InternLM/lmdeploy/issues/2042.)
Could you please explain why W8A8 quantization is not supported for multi-modal models? Is it due to model accuracy concerns?
### Related resources
_No response_
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.