google-deepmind / google-deepmind/limit

"recall_at_2": 0.162 for "sentence-transformers/all-MiniLM-L6-v2" and "LIMITSmallRetrieval"

Open
#14 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
669
Forks
55
PR merge metrics
No merged PRs in 30d

Description

Hi

I have been running MTEB for "sentence-transformers/all-MiniLM-L6-v2" and "LIMITSmallRetrieval" and receive

"recall_at_2": 0.162

I have been running my own code using Apache Lucene Vector Search Implementation for "sentence-transformers/all-MiniLM-L6-v2" and "LIMITSmallRetrieval" and receive

"recall_at_2": 0.129

or when I run it a second time, then I receive

"recall_at_2": 0.134

Apache Lucene is using HNSW graph implementation https://lucene.apache.org/core/9_1_0/core/org/apache/lucene/util/hnsw/HnswGraph.html which quite probably has an effect on the search results, and therefore recall@2 is quite probably different than MTEB, but it would be good to compare this with the results of others.

I could not find anything on https://huggingface.co/spaces/mteb/leaderboard or otherwise.

Where could I find other results to compare my own implementation(s)?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.