Qwen Chat CUDA OutOfMemory
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 275
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 33
Description
RTX 4090 24G,
Qwen-7B-Chat
loads OK:
```
model_config = ModelConfig(lora_infos={
"lora_1": conf['lora_1'],
"lora_2": conf['lora_2'],
})
model = ModelFactory.from_huggingface(conf['base_model_dir'], model_config=model_config)
pipeline = Pipeline(model, model.tokenizer)
```
But the following causes OutOfMemoryError
```
# rtp_sys.conf
#
# [
# {"task_id": 1, "prompt": " <|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>"}
# ]
import os
os.environ['MULTI_TASK_PROMPT'] = './rtp_sys.conf'
model_config = ModelConfig(lora_infos={
"lora_1": conf['lora_1'],
"lora_2": conf['lora_2'],
})
model = ModelFactory.from_huggingface(conf['base_model_dir'], model_config=model_config)
pipeline = Pipeline(model, model.tokenizer)
File "/data1/miniconda/xxx/rtp-llm/lib/python3.10/site-packages/maga_transformer/utils/model_weights_loader.py", line 304, in _load_layer_weight
tensor = self._split_and_sanitize_tensor(tensor, weight).to(device)
torch.cuda.OutOfMemoryError: CUDA out of memory.
```
I've tried with and without `export ENABLE_FMHA=OFF`
I'm referring to [this link SystemPrompt-Tutorial](https://github.com/alibaba/rtp-llm/blob/main/docs/SystemPrompt-Tutorial.md)
For the record, my requirement here are:
1. have 2 LoRAs, and during one round chat I **have to switch** between them
2. I need to use chat interface. Since Qwen does not come with chat_template, I need a way to implement "make_context"
Because of requirement 1, `python3 -m maga_transformer.start_server` + http post with OpenAI request is not the case. (Or if you could switch different adapter for a up running server, please tell me)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with docs/SystemPrompt-Tutorial.md and the reproduction using MULTI_TASK_PROMPT and rtp_sys.conf. Trace model loading through maga_transformer/utils/model_weights_loader.py at _load_layer_weight, then compare the working and failing ModelConfig paths. Done means identifying the cause of the CUDA out-of-memory error and documenting or correcting the supported way to use system prompts with multiple switchable LoRAs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- backend, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100