DAMO-NLP-SG / DAMO-NLP-SG/VideoLLaMA2
Cannot load model in 4bit with VideoLLaMA2.1-7B-AV model
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 90
- PR merge metrics
- No merged PRs in 30d
Description
Hi team,
I am having trouble to load the VideoLLaMA2.1-7B-AV model in 4 bit
Below is the error:
```
Exception has occurred: RuntimeError
Only Tensors created explicitly by the user (graph leaves) support the deepcopy protocol at the moment. If you were attempting to deepcopy a module, this may be because of a torch.nn.utils.weight_norm usage, see https://github.com/pytorch/pytorch/pull/103001
File "/home/ubuntu/workspace/VideoLLaMA2/videollama2/model/__init__.py", line 182, in load_pretrained_model
model = Videollama2Qwen2ForCausalLM.from_pretrained(model_path, low_cpu_mem_usage=True, config=config, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/workspace/VideoLLaMA2/videollama2/__init__.py", line 17, in model_init
tokenizer, model, processor, context_len = load_pretrained_model(model_path, None, model_name, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/workspace/VideoLLaMA2/inference_demo.py", line 10, in inference
model, processor, tokenizer = model_init(model_path, load_4bit=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/workspace/VideoLLaMA2/inference_demo.py", line 66, in
inference(args)
RuntimeError: Only Tensors created explicitly by the user (graph leaves) support the deepcopy protocol at the moment. If you were attempting to deepcopy a module, this may be because of a torch.nn.utils.weight_norm usage, see https://github.com/pytorch/pytorch/pull/103001
```
Appreciate any suggestions.
Thanks!
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.