DAMO-NLP-SG / DAMO-NLP-SG/VideoLLaMA2
About the training efficiency
Open
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 90
- PR merge metrics
- No merged PRs in 30d
Description
Hi, the videoLLama2 got improvement from v1. However, I noticed that there is a little bit high cost.
There is a expanding image into -> videos = [x.unsqueeze(0).expand(num_frames, -1, -1, -1) if modal == 'image' else x for x, modal in zip(images_or_videos, modalities)]
same dimension as video which is b, t, c, h, w
So that, if there were batchSize=10, and numFrames is 8, there should be totally batchSize=80 for vision encoder. Which is extremly **HUGE**.
Why not consider some other way around to avoid this? Maybe do not expanding image?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.