OpenGVLab / OpenGVLab/InternVideo

Clip model size is too small

Open
#114 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2.4k
Forks
160
PR merge metrics
No merged PRs in 30d

Description

Hello, really appreciate for your great work.
https://github.com/OpenGVLab/InternVideo/blob/main/InternVideo2/multi_modality/MODEL_ZOO.md
I checked that you guys wrote "We also learn a CLIP-style InternVideo2 indicated by InternVideo2clip. It is post-pretrained from InternVideo2s2 by only preserving video and text encoders and contrastive loss." in your paper.

But I found out that this model [InternVideo2-CLIP-1B-224p-f8] in huggingface is too small, like just a few MB. And according to the right before issue I noticed that that pth file in huggingface is "add on parameter", not a full parameter.

  1. So as i understood, there might be only clip model that post trained after stage2 right?
  2. I want to know how can I initialize that clip model and utilize. I want to get clip score from that model. It would be really grateful if you let me know exact way to do that. ( It is quite confusing no matter how much time i refer to your readme and demo.ipynb file.)

Thank you in advance.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with InternVideo2/multi_modality/MODEL_ZOO.md and the README and demo.ipynb mentioned in the issue. Trace how the InternVideo2-CLIP-1B-224p-f8 checkpoint is loaded and determine how its add-on parameters relate to the base model. Done means documenting the exact initialization and clip-scoring steps, including which files or weights are required.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook, python
Domain
machine-learning
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.