InternLM / InternLM/InternLM-XComposer
7B版本无法多卡运行
- Dominant language
- Python
- Stars
- 2.9k
- Forks
- 175
- PR merge metrics
- No merged PRs in 30d
Description
使用官方提供的7B版本,单卡24G内存的RTX上无法运行,报OOM错误,指定卡号后无法生效,依然还是只占用第0卡,要怎么推理才可以正常运行
```python
import torch
from transformers import AutoModel, AutoTokenizer
torch.set_grad_enabled(False)
ckpt_path='/home/my/.cache/modelscope/hub/Shanghai_AI_Laboratory/internlm-xcomposer2-vl-7b'
# init model and tokenizer
model = AutoModel.from_pretrained(ckpt_path, trust_remote_code=True).cuda().eval()
tokenizer = AutoTokenizer.from_pretrained(ckpt_path, trust_remote_code=True)
text = '仔细描述这张图'
image='/home/my/cat.jpg'
with torch.cuda.amp.autocast():
response, _ = model.chat(tokenizer, query=text, image=image, history=[], do_sample=False)
print(response)
```
报错:OOM错误

## 代码中指定所有卡号(机器信息:4卡,每张24G内存)
```python
import os
os.environ['CUDA_VISIBLE_DEVICES'] = '0,1,2,3'
import torch
from transformers import AutoModel, AutoTokenizer
torch.set_grad_enabled(False)
ckpt_path='/home/my/.cache/modelscope/hub/Shanghai_AI_Laboratory/internlm-xcomposer2-vl-7b'
# init model and tokenizer
model = AutoModel.from_pretrained(ckpt_path, trust_remote_code=True).cuda().eval()
tokenizer = AutoTokenizer.from_pretrained(ckpt_path, trust_remote_code=True)
text = '仔细描述这张图'
image='/home/my/cat.jpg'
with torch.cuda.amp.autocast():
response, _ = model.chat(tokenizer, query=text, image=image, history=[], do_sample=False)
print(response)
```
还是一样的错误,查看nvidia-smi发现实际还是跑在一张卡上,没有分布到其余卡上
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.