InternLM / InternLM/InternLM-XComposer

显存占用过大疑问

Open
#429 1 comment 0 reactions 1 assignee Claimed by @LightDXY View on GitHub
Dominant language
Python
Stars
2.9k
Forks
175
PR merge metrics
No merged PRs in 30d

Description

使用示例代码进行推理,4bit的在回答的时候单张图像最大达到了38G的显存占用,是否是正常的。无量化的模型版本直接报错了。
from lmdeploy import TurbomindEngineConfig, pipeline
from lmdeploy.vl import load_image
engine_config = TurbomindEngineConfig(model_format='awq')
pipe = pipeline('internlm/internlm-xcomposer2d5-7b-4bit', backend_config=engine_config)
image = load_image('/root/workspace/InternLM-XComposer/examples/cars1.jpg')
response = pipe(('describe this image', image))
print(response.text)

![image](https://github.com/user-attachments/assets/7a79b2bb-d360-4c67-8077-130c5d74315a)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.