OpenGVLab / OpenGVLab/InternVideo
S2 pretrained model of InternVideo2 does not work well for Zero-Shot Video-Text Retrieval
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.4k
- Forks
- 160
- PR merge metrics
- No merged PRs in 30d
Description
直接跑demo/demo.ipynb, 模型选用https://huggingface.co/OpenGVLab/InternVideo2-Stage2_1B-224p-f4/blob/main/InternVideo2-stage2_1b-224p-f4.pt 发现效果不太理想。
首先需要修改两个地方才能正确加载模型：
1、demo/demo.ipynb 中在setup_internvideo2(config)前面加上一句 config['pretrained_path'] = model_pth
2、demo/utils.py 第82和84行改成is_pretrain=True
修改后demo中提供的视频和10个句子的相似度分数（不经过softmax）为：
可以发现分数最高者并不是正确的描述，同时十个句子得分都比较接近。
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the result by running demo/demo.ipynb with the InternVideo2 checkpoint, then inspect setup_internvideo2 and the is_pretrain settings in demo/utils.py around lines 82 and 84. Compare the ten unnormalized similarity scores with the provided video and captions; done means the model loads correctly and ranks the matching description highest with clearly separated scores.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100