DAMO-NLP-SG / DAMO-NLP-SG/VideoLLaMA2

Problem about processor in load_pretrained_model

Open
#74 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.3k
Forks
90
PR merge metrics
No merged PRs in 30d

Description

Hi Teams,

I'm trying to evaluate VideoLLaMA2 on MVBench. As I run the inference_video_mcqa_mvbench.py, the following traceback occurs:

```
Traceback (most recent call last):
File "/***/VideoLLaMA2/videollama2/eval/inference_video_mcqa_mvbench.py", line 203, in
run_inference(args)
File "/***/VideoLLaMA2/videollama2/eval/inference_video_mcqa_mvbench.py", line 164, in run_inference
for i, line in enumerate(tqdm(val_loader)):
File "/***/python3.11/site-packages/tqdm/std.py", line 1178, in __iter__
for obj in iterable:
File "/***/lib/python3.11/site-packages/torch/utils/data/dataloader.py", line 630, in __next__
data = self._next_data()
^^^^^^^^^^^^^^^^^
File "/***/lib/python3.11/site-packages/torch/utils/data/dataloader.py", line 1345, in _next_data
return self._process_data(data)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/***/lib/python3.11/site-packages/torch/utils/data/dataloader.py", line 1371, in _process_data
data.reraise()
File "/***/lib/python3.11/site-packages/torch/_utils.py", line 694, in reraise
raise exception
AttributeError: Caught AttributeError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/***/lib/python3.11/site-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop
data = fetcher.fetch(index)
^^^^^^^^^^^^^^^^^^^^
File "/***/lib/python3.11/site-packages/torch/utils/data/_utils/fetch.py", line 51, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/***/lib/python3.11/site-packages/torch/utils/data/_utils/fetch.py", line 51, in
data = [self.dataset[idx] for idx in possibly_batched_index]
~~~~~~~~~~~~^^^^^
File "/***/VideoLLaMA2/videollama2/eval/inference_video_mcqa_mvbench.py", line 50, in __getitem__
torch_imgs = self.processor(video_path, s=bound[0], e=bound[1])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/***/VideoLLaMA2/./videollama2/mm_utils.py", line 202, in process_video
video = processor.preprocess(images, return_tensors='pt')['pixel_values']
^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'preprocess'
```

I find that the `processor` in
https://github.com/DAMO-NLP-SG/VideoLLaMA2/blob/42bf9fe09656f0a155d96db77178fb74ccc9828d/videollama2/model/__init__.py#L193-L208
is initialized as `None`. For `model_type=mistral` in `config.json` of [VideoLLaMA2-7B](https://huggingface.co/DAMO-NLP-SG/VideoLLaMA2-7B) and [VideoLLaMA2-7B-16F](https://huggingface.co/DAMO-NLP-SG/VideoLLaMA2-7B-16F), the `processor` keeps as None, which may cause the traceback above. Could you please help me address the problem? Thanks!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.