Error when trying to load quantized llava-v1.6-34b

Here's what I've done:
1. Quantized llava-v1.6-34b with the following code:
```python
from llava.model.builder import load_pretrained_model
tokenizer, model, image_processor, context_len = load_pretrained_model(
model_path='/root/ssd/llava-v1.6-34b',
model_base=None,
model_name='llava-v1.6-34b',
load_8bit=False,
load_4bit=True,
device_map='auto',
device='cuda',
use_flash_attn=False
)
tokenizer.save_pretrained('/root/ssd/llava-v1.6-34b-int4')
model.save_pretrained('/root/ssd/llava-v1.6-34b-int4')
```
2. Modified config.json 86L "model_type": "llava_llama" -> "llava".
3. Successfully loaded the quantized model using both:
```python
from llava.model import LlavaLlamaForCausalLM
model = LlavaLlamaForCausalLM.from_pretrained('/root/ssd/llava-v1.6-34b-int4')
```
and
```python
from llava.model.builder import load_pretrained_model
tokenizer, model, image_processor, context_len = load_pretrained_model(
model_path='/root/ssd/llava-v1.6-34b-int4',
model_base=None,
model_name='llava-v1.6-34b',
load_8bit=False,
load_4bit=True,
device_map='auto',
device='cuda',
use_flash_attn=False
)
```
4. Successfully loaded the vanilla model with:
```python
from lmdeploy import pipeline, TurbomindEngineConfig
pipe = pipeline(
model_name='liuhaotian/llava-v1.6-34b',
model_path='/root/ssd/llava-v1.6-34b',
backend_config=TurbomindEngineConfig(
tp=4,
model_format='hf',
session_len=8192,
cache_max_entry_count=0.1,
),
log_level='INFO'
)
```
However, when attempting to load the quantified model as follows, I encounter an error:
```python
pipe = pipeline(
model_name='liuhaotian/llava-v1.6-34b',
model_path='/root/ssd/llava-v1.6-34b-int4', # -int4 here
backend_config=TurbomindEngineConfig(
tp=4,
model_format='hf',
session_len=8192,
cache_max_entry_count=0.1,
),
log_level='INFO'
)
```
Here's the error message:
```shell
2024-04-10 08:09:02,731 - lmdeploy - INFO - Using turbomind engine
2024-04-10 08:09:02,731 - lmdeploy - INFO - input backend=turbomind, backend_config=TurbomindEngineConfig(model_name=None, model_format='hf', tp=4, session_len=8192, max_batch_size=128, cache_max_entry_count=0.1, cache_block_seq_len=64, quant_policy=0, rope_scaling_factor=0.0, use_logn_attn=False, download_dir=None, revision=None, max_prefill_token_num=8192)
2024-04-10 08:09:02,731 - lmdeploy - INFO - input chat_template_config=None
2024-04-10 08:09:02,731 - lmdeploy - WARNING - Could not find liuhaotian/llava-v1.6-34b-int4 in registered models. Register liuhaotian/llava-v1.6-34b-int4 using the BaseChatTemplate.
2024-04-10 08:09:02,731 - lmdeploy - INFO - updated chat_template_onfig=ChatTemplateConfig(model_name='liuhaotian/llava-v1.6-34b-int4', system=None, meta_instruction=None, eosys=None, user=None, eoh=None, assistant=None, eoa=None, separator=None, capability=None, stop_words=None)
2024-04-10 08:09:02,781 - lmdeploy - WARNING - model_source: hf_model
2024-04-10 08:09:05,017 - lmdeploy - WARNING - model_config:
[llama]
model_name = base
tensor_para_size = 4
head_num = 56
kv_head_num = 8
vocab_size = 64000
num_layer = 60
inter_size = 73400320
norm_eps = 1e-05
attn_bias = 0
start_id = 64000
end_id = 7
session_len = 8192
weight_type = fp16
rotary_embedding = 128
rope_theta = 5000000.0
size_per_head = 128
group_size = 0
max_batch_size = 128
max_context_token_num = 1
step_length = 1
cache_max_entry_count = 0.1
cache_block_seq_len = 64
cache_chunk_size = -1
num_tokens_per_iter = 8192
max_prefill_iters = 1
extra_tokens_per_iter = 0
use_context_fmha = 1
quant_policy = 0
max_position_embeddings = 4096
rope_scaling_factor = 0.0
use_dynamic_ntk = 0
use_logn_attn = 0
[TM][INFO] Set logger level by INFO
[TM][WARNING] [LlamaTritonModel] `max_context_token_num` = 8192.
Exception in thread Thread-6 (_create_weight_func):
Traceback (most recent call last):
File "/root/miniconda3/envs/LMdeploy/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/root/miniconda3/envs/LMdeploy/lib/python3.10/threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "/root/miniconda3/envs/LMdeploy/lib/python3.10/site-packages/lmdeploy/turbomind/turbomind.py", line 196, in _create_weight_func
model_comm.create_shared_weights(device_id, rank)
RuntimeError: [TM][ERROR] CUDA runtime error: out of memory /lmdeploy/src/turbomind/utils/memory_utils.cu:32
Exception in thread Thread-7 (_create_weight_func):
Traceback (most recent call last):
File "/root/miniconda3/envs/LMdeploy/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/root/miniconda3/envs/LMdeploy/lib/python3.10/threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "/root/miniconda3/envs/LMdeploy/lib/python3.10/site-packages/lmdeploy/turbomind/turbomind.py", line 196, in _create_weight_func
model_comm.create_shared_weights(device_id, rank)
RuntimeError: [TM][ERROR] CUDA runtime error: out of memory /lmdeploy/src/turbomind/utils/memory_utils.cu:32
Exception in thread Thread-4 (_create_weight_func):
Traceback (most recent call last):
File "/root/miniconda3/envs/LMdeploy/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/root/miniconda3/envs/LMdeploy/lib/python3.10/threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "/root/miniconda3/envs/LMdeploy/lib/python3.10/site-packages/lmdeploy/turbomind/turbomind.py", line 196, in _create_weight_func
model_comm.create_shared_weights(device_id, rank)
RuntimeError: [TM][ERROR] CUDA runtime error: out of memory /lmdeploy/src/turbomind/utils/memory_utils.cu:32
Exception in thread Thread-5 (_create_weight_func):
Traceback (most recent call last):
File "/root/miniconda3/envs/LMdeploy/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/root/miniconda3/envs/LMdeploy/lib/python3.10/threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "/root/miniconda3/envs/LMdeploy/lib/python3.10/site-packages/lmdeploy/turbomind/turbomind.py", line 196, in _create_weight_func
model_comm.create_shared_weights(device_id, rank)
RuntimeError: [TM][ERROR] CUDA runtime error: out of memory /lmdeploy/src/turbomind/utils/memory_utils.cu:32
Exception in thread Thread-8 (_get_params):
Traceback (most recent call last):
File "/root/miniconda3/envs/LMdeploy/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
Exception in thread Thread-9 (_get_params):
Traceback (most recent call last):
File "/root/miniconda3/envs/LMdeploy/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
Exception in thread Thread-10 (_get_params):
Traceback (most recent call last):
File "/root/miniconda3/envs/LMdeploy/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/root/miniconda3/envs/LMdeploy/lib/python3.10/threading.py", line 953, in run
Exception in thread self._target(*self._args, **self._kwargs)
File "/root/miniconda3/envs/LMdeploy/lib/python3.10/site-packages/lmdeploy/turbomind/turbomind.py", line 226, in _get_params
self.run()
File "/root/miniconda3/envs/LMdeploy/lib/python3.10/threading.py", line 953, in run
Thread-11 (_get_params):
Traceback (most recent call last):
File "/root/miniconda3/envs/LMdeploy/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run() out = model_comm.get_params(device_id, rank)
self._target(*self._args, **self._kwargs)
File "/root/miniconda3/envs/LMdeploy/lib/python3.10/site-packages/lmdeploy/turbomind/turbomind.py", line 226, in _get_params
RuntimeError: [TM][ERROR] Assertion fail: /lmdeploy/src/turbomind/triton_backend/llama/LlamaTritonModel.cc:384
File "/root/miniconda3/envs/LMdeploy/lib/python3.10/threading.py", line 953, in run
self.run()
File "/root/miniconda3/envs/LMdeploy/lib/python3.10/threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "/root/miniconda3/envs/LMdeploy/lib/python3.10/site-packages/lmdeploy/turbomind/turbomind.py", line 226, in _get_params
out = model_comm.get_params(device_id, rank)self._target(*self._args, **self._kwargs)
File "/root/miniconda3/envs/LMdeploy/lib/python3.10/site-packages/lmdeploy/turbomind/turbomind.py", line 226, in _get_params
out = model_comm.get_params(device_id, rank)
RuntimeError
RuntimeError: [TM][ERROR] Assertion fail: /lmdeploy/src/turbomind/triton_backend/llama/LlamaTritonModel.cc:384
: [TM][ERROR] Assertion fail: /lmdeploy/src/turbomind/triton_backend/llama/LlamaTritonModel.cc:384
out = model_comm.get_params(device_id, rank)
RuntimeError: [TM][ERROR] Assertion fail: /lmdeploy/src/turbomind/triton_backend/llama/LlamaTritonModel.cc:384
```
Despite the error, the GPU memory usage appears to be low (286MiB/22GiB)
And this is my pip list:
```shell
Package Version
------------------------- -----------
accelerate 0.21.0
addict 2.4.0
aiofiles 23.2.1
altair 5.3.0
annotated-types 0.6.0
anyio 4.3.0
attrs 23.2.0
bitsandbytes 0.43.0
certifi 2024.2.2
charset-normalizer 3.3.2
click 8.1.7
contourpy 1.2.1
cycler 0.12.1
einops 0.6.1
einops-exts 0.0.4
exceptiongroup 1.2.0
fastapi 0.110.1
ffmpy 0.3.2
filelock 3.13.3
fire 0.6.0
fonttools 4.50.0
fsspec 2024.3.1
gradio 4.16.0
gradio_client 0.8.1
h11 0.14.0
httpcore 0.17.3
httpx 0.24.0
huggingface-hub 0.22.2
idna 3.6
importlib_metadata 7.1.0
importlib_resources 6.4.0
Jinja2 3.1.3
joblib 1.3.2
jsonschema 4.21.1
jsonschema-specifications 2023.12.1
kiwisolver 1.4.5
llava 1.2.2.post1
lmdeploy 0.3.0
markdown-it-py 3.0.0
markdown2 2.4.13
MarkupSafe 2.1.5
matplotlib 3.8.4
mdurl 0.1.2
mmengine-lite 0.10.3
mpmath 1.3.0
networkx 3.2.1
numpy 1.26.4
nvidia-cublas-cu12 12.1.3.1
nvidia-cuda-cupti-cu12 12.1.105
nvidia-cuda-nvrtc-cu12 12.1.105
nvidia-cuda-runtime-cu12 12.1.105
nvidia-cudnn-cu12 8.9.2.26
nvidia-cufft-cu12 11.0.2.54
nvidia-curand-cu12 10.3.2.106
nvidia-cusolver-cu12 11.4.5.107
nvidia-cusparse-cu12 12.1.0.106
nvidia-nccl-cu12 2.18.1
nvidia-nvjitlink-cu12 12.4.99
nvidia-nvtx-cu12 12.1.105
orjson 3.10.0
packaging 24.0
pandas 2.2.1
peft 0.9.0
pillow 10.3.0
pip 23.3.1
platformdirs 4.2.0
protobuf 5.26.1
psutil 5.9.8
pydantic 2.6.4
pydantic_core 2.16.3
pydub 0.25.1
Pygments 2.17.2
pynvml 11.5.0
pyparsing 3.1.2
python-dateutil 2.9.0.post0
python-multipart 0.0.9
pytz 2024.1
PyYAML 6.0.1
referencing 0.34.0
regex 2023.12.25
requests 2.31.0
rich 13.7.1
rpds-py 0.18.0
ruff 0.3.5
safetensors 0.4.2
scikit-learn 1.2.2
scipy 1.13.0
semantic-version 2.10.0
sentencepiece 0.1.99
setuptools 68.2.2
shellingham 1.5.4
shortuuid 1.0.13
six 1.16.0
sniffio 1.3.1
starlette 0.37.2
svgwrite 1.4.3
sympy 1.12
termcolor 2.4.0
threadpoolctl 3.4.0
tiktoken 0.6.0
timm 0.6.13
tokenizers 0.15.1
tomli 2.0.1
tomlkit 0.12.0
toolz 0.12.1
torch 2.1.2
torchvision 0.16.2
tqdm 4.66.2
transformers 4.37.2
triton 2.1.0
typer 0.12.0
typer-cli 0.12.0
typer-slim 0.12.0
typing_extensions 4.10.0
tzdata 2024.1
urllib3 2.2.1
uvicorn 0.29.0
wavedrom 2.0.3.post3
websockets 11.0.3
wheel 0.41.2
yapf 0.40.2
zipp 3.18.1
```
Thanks a lot for your help!