Run benchmarking with the supported models on BEIR MSMARCO
- Dominant language
- Rust
- Stars
- 244
- Forks
- 29
- Avg merge
- 2d 41m
- Merged PRs (30d)
- 19
Description
We need to run benchmarking on the [BEIR MSMARCO](https://paperswithcode.com/dataset/beir) dataset, to have a better understanding of how the models are performing for retrieval tasks.
We can use the `test` split available on Hugging Face hub:
[QRels](https://huggingface.co/datasets/BeIR/msmarco-qrels)
[Corpus](https://huggingface.co/datasets/BeIR/msmarco)
Proposed metrics:
- NDCG@10
- Precision@10
- Recall@100
Considering non-judged documents as non-relevant.
Contributor guide
Research direction
Start by reviewing the repository's existing benchmarking entry point and supported-model list, then use the BEIR MSMARCO test split from the linked Hugging Face QRels and Corpus datasets. The work is done when benchmarking results report NDCG@10, Precision@10, and Recall@100, treating non-judged documents as non-relevant.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- huggingface
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100