DAMO-NLP-SG / DAMO-NLP-SG/VideoLLaMA2

Reproduce the pre-training tasks of Video-LLaMAv2, but the video dimensions are misaligned.

Open
#141 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.3k
Forks
90
PR merge metrics
No merged PRs in 30d

Description

I am attempting to reproduce the pre-training tasks of Video-LLaMAv2. I have already downloaded the Vallay and LLaVA-image datasets and started experimenting with pre-training. However, I noticed that the video dimensions obtained in `LazySupervisedDataset` and `DataCollatorForSupervisedDataset` are 16, 3, 336, 336. Without making any modifications, I found that the video dimensions became 2, 3, 336, 336 in the forward method of `VideoLLaMA2MistralForCausalLM`. I couldn't find where the changes occurred and couldn't understand the logic behind the modification. Could you help me resolve this issue?
image
image

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.