DAMO-NLP-SG / DAMO-NLP-SG/VideoLLaMA2

Forward pass of the model - how to pass videos?

Open
#105 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.3k
Forks
90
PR merge metrics
No merged PRs in 30d

Description

I see this is how you run inference on the model:

```
output = model.generate(
input_ids,
return_dict_in_generate=True,
output_scores=True,
images_or_videos=videos,
modal_list=['video'],
max_new_tokens=128,
do_sample=True,
temperature=0.2,
use_cache=False,
)
```

I wanted to know how to run training code. The forward pass of the model does not support `modal_list` and `images_or_videos` - then how do I pass the videos? I want to be able to pass `labels` as I need to calculate probabilities. (so I need logits)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the model's generate call and forward pass implementation, focusing on how images_or_videos, modal_list, labels, and logits are handled. Determine whether video inputs and label-based probability calculation are supported in the training path; done means a documented or working forward-pass path that accepts videos and produces logits for labels.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.