DAMO-NLP-SG / DAMO-NLP-SG/VideoLLaMA3

evaluate the 2b and 7b model on MLVU or longVideoBench

Open
#68 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
1.2k
Forks
89
PR merge metrics
No merged PRs in 30d

Description

I evaluated VideoLLama3-2B and VideoLLama3-7B models on MLVU using lmms-eval. All generation settings followed lmms-eval's default parameters. The evaluation results yielded the following metrics:
```
"mlvu_dev": {
"alias": "mlvu_dev",
"mlvu_perception_score,none": 60.407165079699325,
"mlvu_perception_score_stderr,none": "N/A"
}
```
These results differ significantly from those reported in the paper. What factors might have caused this discrepancy?

I also attempted to test MLVU and LongVideoBench using the VideoLlama3 repository, but I encountered the same bugs.

when i run LongVideobenchmark evaluation, I encounter this error:
```
ile "/home/zzz/code/VideoLLaMA3/./evaluation/benchmarks/longvideobench.py", line 172, in generate_instruction
inputs = self.processor(conversation=conversation, return_tensors="pt"),
File "/home/zzz/.cache/huggingface/modules/transformers_modules/VideoLLaMA3-2B/processing_videollama3.py", line 707, in __call__
return self._process_conversation(conversation, images, return_labels, **kwargs)
File "/home/zzz/.cache/huggingface/modules/transformers_modules/VideoLLaMA3-2B/processing_videollama3.py", line 614, in _process_conversation
images = self._gather_multimodal_data(conversation)
File "/home/zzz/.cache/huggingface/modules/transformers_modules/VideoLLaMA3-2B/processing_videollama3.py", line 496, in _gather_multimodal_data
video = content["video"]
KeyError: 'video'
```

when i run MLVU benchmark evaluation, I encounter this error:
```
Traceback (most recent call last):
File "/home/zzz/code/VideoLLaMA3/evaluation/evaluate.py", line 108, in main
response = mm_infer(
File "/home/zzz/code/VideoLLaMA3/./evaluation/register.py", line 52, in mm_infer
output_ids = model.generate(
File "/home/zzz/anaconda3/envs/video_llm/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/home/zzz/.cache/huggingface/modules/transformers_modules/VideoLLaMA3-2B/modeling_videollama3.py", line 426, in generate
) = self.prepare_inputs_labels_for_multimodal(
File "/home/zzz/.cache/huggingface/modules/transformers_modules/VideoLLaMA3-2B/modeling_videollama3.py", line 290, in prepare_inputs_labels_for_multimodal
mm_features = self._get_valid_visual_tokens(mm_features, batched_num_patches, modals)
File "/home/zzz/.cache/huggingface/modules/transformers_modules/VideoLLaMA3-2B/modeling_videollama3.py", line 155, in _get_valid_visual_tokens
mm_features = mm_features[torch.cat(valid_masks)]
IndexError: The shape of the mask [14976] at index 0 does not match the shape of the indexed tensor [15360, 1536] at index 0
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.