DAMO-NLP-SG / DAMO-NLP-SG/VideoLLaMA2

Cannot load model in 4bit with VideoLLaMA2.1-7B-AV model

Open
#151 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.3k
Forks
90
PR merge metrics
No merged PRs in 30d

Description

Hi team,

I am having trouble to load the VideoLLaMA2.1-7B-AV model in 4 bit
Below is the error:

```
Exception has occurred: RuntimeError
Only Tensors created explicitly by the user (graph leaves) support the deepcopy protocol at the moment. If you were attempting to deepcopy a module, this may be because of a torch.nn.utils.weight_norm usage, see https://github.com/pytorch/pytorch/pull/103001
File "/home/ubuntu/workspace/VideoLLaMA2/videollama2/model/__init__.py", line 182, in load_pretrained_model
model = Videollama2Qwen2ForCausalLM.from_pretrained(model_path, low_cpu_mem_usage=True, config=config, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/workspace/VideoLLaMA2/videollama2/__init__.py", line 17, in model_init
tokenizer, model, processor, context_len = load_pretrained_model(model_path, None, model_name, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/workspace/VideoLLaMA2/inference_demo.py", line 10, in inference
model, processor, tokenizer = model_init(model_path, load_4bit=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/workspace/VideoLLaMA2/inference_demo.py", line 66, in
inference(args)
RuntimeError: Only Tensors created explicitly by the user (graph leaves) support the deepcopy protocol at the moment. If you were attempting to deepcopy a module, this may be because of a torch.nn.utils.weight_norm usage, see https://github.com/pytorch/pytorch/pull/103001
```
Appreciate any suggestions.

Thanks!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.