DAMO-NLP-SG / DAMO-NLP-SG/VideoLLaMA2
Problem about processor in load_pretrained_model
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 90
- PR merge metrics
- No merged PRs in 30d
Description
Hi Teams,
I'm trying to evaluate VideoLLaMA2 on MVBench. As I run the inference_video_mcqa_mvbench.py, the following traceback occurs:
```
Traceback (most recent call last):
File "/***/VideoLLaMA2/videollama2/eval/inference_video_mcqa_mvbench.py", line 203, in
run_inference(args)
File "/***/VideoLLaMA2/videollama2/eval/inference_video_mcqa_mvbench.py", line 164, in run_inference
for i, line in enumerate(tqdm(val_loader)):
File "/***/python3.11/site-packages/tqdm/std.py", line 1178, in __iter__
for obj in iterable:
File "/***/lib/python3.11/site-packages/torch/utils/data/dataloader.py", line 630, in __next__
data = self._next_data()
^^^^^^^^^^^^^^^^^
File "/***/lib/python3.11/site-packages/torch/utils/data/dataloader.py", line 1345, in _next_data
return self._process_data(data)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/***/lib/python3.11/site-packages/torch/utils/data/dataloader.py", line 1371, in _process_data
data.reraise()
File "/***/lib/python3.11/site-packages/torch/_utils.py", line 694, in reraise
raise exception
AttributeError: Caught AttributeError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/***/lib/python3.11/site-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop
data = fetcher.fetch(index)
^^^^^^^^^^^^^^^^^^^^
File "/***/lib/python3.11/site-packages/torch/utils/data/_utils/fetch.py", line 51, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/***/lib/python3.11/site-packages/torch/utils/data/_utils/fetch.py", line 51, in
data = [self.dataset[idx] for idx in possibly_batched_index]
~~~~~~~~~~~~^^^^^
File "/***/VideoLLaMA2/videollama2/eval/inference_video_mcqa_mvbench.py", line 50, in __getitem__
torch_imgs = self.processor(video_path, s=bound[0], e=bound[1])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/***/VideoLLaMA2/./videollama2/mm_utils.py", line 202, in process_video
video = processor.preprocess(images, return_tensors='pt')['pixel_values']
^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'preprocess'
```
I find that the `processor` in
https://github.com/DAMO-NLP-SG/VideoLLaMA2/blob/42bf9fe09656f0a155d96db77178fb74ccc9828d/videollama2/model/__init__.py#L193-L208
is initialized as `None`. For `model_type=mistral` in `config.json` of [VideoLLaMA2-7B](https://huggingface.co/DAMO-NLP-SG/VideoLLaMA2-7B) and [VideoLLaMA2-7B-16F](https://huggingface.co/DAMO-NLP-SG/VideoLLaMA2-7B-16F), the `processor` keeps as None, which may cause the traceback above. Could you please help me address the problem? Thanks!
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.