TIGER-AI-Lab / TIGER-AI-Lab/VLM2Vec
More Retriever Implementations for Latest Multi-Vector Models
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 684
- Forks
- 64
- Avg merge
- 8m
- Merged PRs (30d)
- 1
Description
Hi VLM2Vec Team @memray @bryant1410 @wenhuchen ,
Thanks for your great work VLM2Vec-V2 model and MMEB pipeline! I found there are some latest multi-vector models with good performance in the popular vidore-benchmark-v2 (https://huggingface.co/spaces/vidore/vidore-leaderboard) as follows. I believe they also deserve the evaluation on your MMEB benchmark:
- vidore/colqwen2.5-v0.2 (https://huggingface.co/vidore/colqwen2.5-v0.2)
- nvidia/llama-nemoretriever-colembed-3b-v1 (https://huggingface.co/nvidia/llama-nemoretriever-colembed-3b-v1)
- nomic-ai/colnomic-embed-multimodal-3b (https://huggingface.co/nomic-ai/colnomic-embed-multimodal-3b)
I wonder if you can implement these retrievers based on your VLM2Vec-V2 codebase and evaluate them. The community will look forward to this. Thanks!
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the VLM2Vec-V2 codebase and its MMEB benchmark entry points to understand how retrievers are implemented and evaluated. Add support for vidore/colqwen2.5-v0.2, nvidia/llama-nemoretriever-colembed-3b-v1, and nomic-ai/colnomic-embed-multimodal-3b, then run the benchmark and confirm evaluation results for all three models.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- huggingface, python
- Domain
- machine-learning, search
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100