InternLM / InternLM/lmdeploy

[Feature] Will multi-modal models support W8A8 quantization in the future?

Open
#2,496 6 comments 0 reactions 1 assignee Claimed by @AllentDan View on GitHub
Dominant language
Python
Stars
8.1k
Forks
748
Avg merge
6d 2h
Merged PRs (30d)
54

Description

### Motivation

Our business model (Internvl 2-26B) outputs very few tokens (1-2 tokens) after prompt optimization, which can be considered as only the prefill stage. Therefore, we hope to use W8A8 quantization to speed up the inference process. However, we found that lmdeploy does not support W8A8 inference for multi-modal models: [https://github.com/InternLM/lmdeploy/issues/2042.](https://github.com/InternLM/lmdeploy/issues/2042.)

Could you please explain why W8A8 quantization is not supported for multi-modal models? Is it due to model accuracy concerns?

### Related resources

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.