DAMO-NLP-SG / DAMO-NLP-SG/VideoLLaMA2
Forward pass of the model - how to pass videos?
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 90
- PR merge metrics
- No merged PRs in 30d
Description
I see this is how you run inference on the model:
```
output = model.generate(
input_ids,
return_dict_in_generate=True,
output_scores=True,
images_or_videos=videos,
modal_list=['video'],
max_new_tokens=128,
do_sample=True,
temperature=0.2,
use_cache=False,
)
```
I wanted to know how to run training code. The forward pass of the model does not support `modal_list` and `images_or_videos` - then how do I pass the videos? I want to be able to pass `labels` as I need to calculate probabilities. (so I need logits)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the model's generate call and forward pass implementation, focusing on how images_or_videos, modal_list, labels, and logits are handled. Determine whether video inputs and label-based probability calculation are supported in the training path; done means a documented or working forward-pass path that accepts videos and produces logits for labels.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100