alibaba / alibaba/rtp-llm

Qwen Chat CUDA OutOfMemory

Open
#63 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.3k
Forks
275
Avg merge
3d 17h
Merged PRs (30d)
33

Description

RTX 4090 24G,
Qwen-7B-Chat

loads OK:
```
model_config = ModelConfig(lora_infos={
"lora_1": conf['lora_1'],
"lora_2": conf['lora_2'],
})
model = ModelFactory.from_huggingface(conf['base_model_dir'], model_config=model_config)
pipeline = Pipeline(model, model.tokenizer)
```

But the following causes OutOfMemoryError
```
# rtp_sys.conf
#
# [
# {"task_id": 1, "prompt": " <|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>"}
# ]

import os
os.environ['MULTI_TASK_PROMPT'] = './rtp_sys.conf'
model_config = ModelConfig(lora_infos={
"lora_1": conf['lora_1'],
"lora_2": conf['lora_2'],
})
model = ModelFactory.from_huggingface(conf['base_model_dir'], model_config=model_config)
pipeline = Pipeline(model, model.tokenizer)

File "/data1/miniconda/xxx/rtp-llm/lib/python3.10/site-packages/maga_transformer/utils/model_weights_loader.py", line 304, in _load_layer_weight
tensor = self._split_and_sanitize_tensor(tensor, weight).to(device)
torch.cuda.OutOfMemoryError: CUDA out of memory.
```

I've tried with and without `export ENABLE_FMHA=OFF`
I'm referring to [this link SystemPrompt-Tutorial](https://github.com/alibaba/rtp-llm/blob/main/docs/SystemPrompt-Tutorial.md)

For the record, my requirement here are:
1. have 2 LoRAs, and during one round chat I **have to switch** between them
2. I need to use chat interface. Since Qwen does not come with chat_template, I need a way to implement "make_context"

Because of requirement 1, `python3 -m maga_transformer.start_server` + http post with OpenAI request is not the case. (Or if you could switch different adapter for a up running server, please tell me)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with docs/SystemPrompt-Tutorial.md and the reproduction using MULTI_TASK_PROMPT and rtp_sys.conf. Trace model loading through maga_transformer/utils/model_weights_loader.py at _load_layer_weight, then compare the working and failing ModelConfig paths. Done means identifying the cause of the CUDA out-of-memory error and documenting or correcting the supported way to use system prompts with multiple switchable LoRAs.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
backend, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.