InternLM / InternLM/InternLM-XComposer

7B版本无法多卡运行

Open
#303 5 comments 0 reactions 1 assignee Claimed by @myownskyW7 View on GitHub
Dominant language
Python
Stars
2.9k
Forks
175
PR merge metrics
No merged PRs in 30d

Description

使用官方提供的7B版本,单卡24G内存的RTX上无法运行,报OOM错误,指定卡号后无法生效,依然还是只占用第0卡,要怎么推理才可以正常运行
```python
import torch
from transformers import AutoModel, AutoTokenizer

torch.set_grad_enabled(False)
ckpt_path='/home/my/.cache/modelscope/hub/Shanghai_AI_Laboratory/internlm-xcomposer2-vl-7b'

# init model and tokenizer
model = AutoModel.from_pretrained(ckpt_path, trust_remote_code=True).cuda().eval()
tokenizer = AutoTokenizer.from_pretrained(ckpt_path, trust_remote_code=True)

text = '仔细描述这张图'
image='/home/my/cat.jpg'
with torch.cuda.amp.autocast():
response, _ = model.chat(tokenizer, query=text, image=image, history=[], do_sample=False)
print(response)
```
报错:OOM错误
![tmp](https://github.com/InternLM/InternLM-XComposer/assets/52484098/1be54648-609e-498d-8647-43589901887b)

## 代码中指定所有卡号(机器信息:4卡,每张24G内存)
```python
import os

os.environ['CUDA_VISIBLE_DEVICES'] = '0,1,2,3'

import torch
from transformers import AutoModel, AutoTokenizer

torch.set_grad_enabled(False)
ckpt_path='/home/my/.cache/modelscope/hub/Shanghai_AI_Laboratory/internlm-xcomposer2-vl-7b'

# init model and tokenizer
model = AutoModel.from_pretrained(ckpt_path, trust_remote_code=True).cuda().eval()
tokenizer = AutoTokenizer.from_pretrained(ckpt_path, trust_remote_code=True)

text = '仔细描述这张图'
image='/home/my/cat.jpg'
with torch.cuda.amp.autocast():
response, _ = model.chat(tokenizer, query=text, image=image, history=[], do_sample=False)
print(response)
```

还是一样的错误,查看nvidia-smi发现实际还是跑在一张卡上,没有分布到其余卡上

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.