InternLM / InternLM/InternLM-XComposer
显存占用过大疑问
Open
- Dominant language
- Python
- Stars
- 2.9k
- Forks
- 175
- PR merge metrics
- No merged PRs in 30d
Description
使用示例代码进行推理,4bit的在回答的时候单张图像最大达到了38G的显存占用,是否是正常的。无量化的模型版本直接报错了。
from lmdeploy import TurbomindEngineConfig, pipeline
from lmdeploy.vl import load_image
engine_config = TurbomindEngineConfig(model_format='awq')
pipe = pipeline('internlm/internlm-xcomposer2d5-7b-4bit', backend_config=engine_config)
image = load_image('/root/workspace/InternLM-XComposer/examples/cars1.jpg')
response = pipe(('describe this image', image))
print(response.text)

Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.