DAMO-NLP-SG / DAMO-NLP-SG/VideoLLaMA2
Reproduce the pre-training tasks of Video-LLaMAv2, but the video dimensions are misaligned.
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 90
- PR merge metrics
- No merged PRs in 30d
Description
I am attempting to reproduce the pre-training tasks of Video-LLaMAv2. I have already downloaded the Vallay and LLaVA-image datasets and started experimenting with pre-training. However, I noticed that the video dimensions obtained in `LazySupervisedDataset` and `DataCollatorForSupervisedDataset` are 16, 3, 336, 336. Without making any modifications, I found that the video dimensions became 2, 3, 336, 336 in the forward method of `VideoLLaMA2MistralForCausalLM`. I couldn't find where the changes occurred and couldn't understand the logic behind the modification. Could you help me resolve this issue?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.