deven96 / deven96/ahnlich

Run benchmarking with the supported models on BEIR MSMARCO

Open
#177 1 comment 0 reactions 0 assignees View on GitHub
evaluation help wanted
Dominant language
Rust
Stars
244
Forks
29
Avg merge
2d 41m
Merged PRs (30d)
19

Description

We need to run benchmarking on the [BEIR MSMARCO](https://paperswithcode.com/dataset/beir) dataset, to have a better understanding of how the models are performing for retrieval tasks.

We can use the `test` split available on Hugging Face hub:

[QRels](https://huggingface.co/datasets/BeIR/msmarco-qrels)
[Corpus](https://huggingface.co/datasets/BeIR/msmarco)

Proposed metrics:

- NDCG@10
- Precision@10
- Recall@100

Considering non-judged documents as non-relevant.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the repository's existing benchmarking entry point and supported-model list, then use the BEIR MSMARCO test split from the linked Hugging Face QRels and Corpus datasets. The work is done when benchmarking results report NDCG@10, Precision@10, and Recall@100, treating non-judged documents as non-relevant.

Written by the indexing model from the issue text.

Assessment

Tech stack
huggingface
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.