OpenGVLab / OpenGVLab/Ask-Anything
Evaluation results on MVBench different from the paper
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 3.4k
- Forks
- 268
- PR merge metrics
- No merged PRs in 30d
Description
Hi, I have tested the VideoChat2 model on my server and found that the test results are different from the paper. My results are listed as follows:
{"Action Sequence": 66.0, "Action Prediction": 47.5, "Action Antonym": 83.5, "Fine-grained Action": 49.5, "Unexpected Action": 60.0, "Object Existence": 57.99999999999999, "Object Interaction": 71.5, "Object Shuffle": 41.5, "Moving Direction": 23.0, "Action Localization": 22.5, "Scene Transition": 88.5, "Action Count": 39.5, "Moving Count": 42.0, "Moving Attribute": 58.5, "State Change": 44.0, "Fine-grained Pose": 49.0, "Character Order": 36.5, "Egocentric Navigation": 35.0, "Episodic Reasoning": 38.5, "Counterfactual Inference": 65.0, "Avg": 50.975}
The results for OS, AL, AC, ER, and CI are different. Could you help me find the reasons?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the VideoChat2 evaluation on MVBench and comparing the listed results with the paper. Focus on the reported differences for OS, AL, AC, ER, and CI, and document the evaluation conditions and the identified reason for the mismatch as the completion criteria.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100