OpenGVLab / OpenGVLab/InternVideo
internvideo2中文支持，如何正确调用internvideo2_clip模型
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.4k
- Forks
- 160
- PR merge metrics
- No merged PRs in 30d
Description
想要在自有中文检索数据集上应用internvideo2，看了下最接近的是vatex_cn中使用internvideo2_clip模型，但是在加载过程中遇到些疑问。
- 共计需要加载哪些ckpt？
目前看是这几个组件
chinese_alpaca_lora_7b
InternVideo2-stage2_1b-224p-f4.pt
1B_clip.pth
internvl_c_13b_224px.pth
配置文件如下
tokenizer_path="chinese_alpaca_lora_7b",
vision_ckpt_path="OpenGVLab__InternVideo2-Stage2_1B-224p-f4/InternVideo2-stage2_1b-224p-f4.pt",
load_vision_ckpt_from_internvideo2_stage2=True,
text_ckpt_path="internvl_c_13b_224px.pth"
extra_ckpt_path="OpenGVLab__InternVideo2-CLIP-1B-224p-f8/1B_clip.pth"
但是找不到可以配置1B_clip.pth的地方，在demo中直接在config中设置pretrained_path还是在配置文件中设置extra_ckpt_path?
2.加载ckpt的代码不知道是不是有问题，我最终产出的排序是这样的，不太符合预期。目前使用的是在配置文件中设置extra_ckpt_path，也就是1里的配置
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the vatex_cn example and its demo/config to trace how chinese_alpaca_lora_7b, InternVideo2-stage2_1b-224p-f4.pt, 1B_clip.pth, and internvl_c_13b_224px.pth are loaded. Compare the configured extra_ckpt_path and pretrained_path usage, then verify the resulting ranking on a Chinese retrieval dataset. Done means the checkpoint configuration is unambiguous and the loading path produces expected retrieval results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 20/100