DAMO-NLP-SG / DAMO-NLP-SG/VideoLLaMA3

Why does the VideoLLaMA3 demo webpage test exhibit output errors in the time granularity of video appearance?

Open
#30 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
1.2k
Forks
89
PR merge metrics
No merged PRs in 30d

Description

Thank you very much for your open-source contribution, but when I tested the second dog.mp4 video on the website https://huggingface.co/spaces/lixin4ever/VideoLLaMA3, I encountered significant output errors in the time granularity of video appearance. Could you please tell me what might be causing this issue and how to resolve it?

This is my conversation test:
Q:
1.How long is this video in seconds?
2. How many times do dog scenes appear in the video?
3. How many dogs appear each time, and from which seconds to which seconds do they appear, respectively?
A:
1. The video is 8 seconds long.
2. The dog scenes appear three times in the video.
3. Each time, there are four dogs. The first dog scene appears at second 0, the second dog scene appears at second 1, and the third dog scene appears at second 2.

However,the correct answers are:
1. The video is 10 seconds long.
2. Dog scenes appear 3 times in the video.
3. From 2 to 3 seconds, there are 5 dogs; from 3 to 4 seconds, there are 4 dogs; and from 9 to 10 seconds, there are 2 dogs.

The demo performance differs from what is described in the paper. Then,I will deploy VideoLLaMA3 locally to further evaluate its capability in understanding video granularity.

![Image](https://github.com/user-attachments/assets/db625587-6181-4e06-9072-c33b2bdfcb82)

Contributor guide

No contributing guide indexed for this repository

Research direction

The report names the second dog.mp4 video in the VideoLLaMA3 Hugging Face demo; begin by reproducing the three questions there and compare its 8-second response with the stated 10-second ground truth. Then compare the demo behavior with the paper and a local deployment, and document the cause and resolution for the reported timestamp and count errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
huggingface, jupyter-notebook
Domain
ai, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.