Azure / Azure/azure-search-vector-samples

How to Calculate Search Scores and Rankings for Hybrid Searches

Open
#48 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
910
Forks
377
PR merge metrics
No merged PRs in 30d

Description

Hi team,
I am wondering how search scores and rankings are calculated for hybrid searches.
I did a search with keyword search(top=10) and vector search(vector.k=10) and got the following ranking.
RRFscore(d) = Σ (1 / (_k_ + rank_d_i)) with constant _k_ = 59.

**1, Simple Vector Search Result**
| FileName | Vector-Rank | Vector-RRF |
| ------------- | ------------- |------------- |
| A.txt | 2 | 1/(_k_+2)=0.016393443 |
| B.txt | 1 | 1/(_k_+1)=0.016666667 |
| C.txt | 3 | 1/(_k_+3)=0.016129032 |

**2, Simple Keyword Search Result**
| FileName | Keyword-Rank | Keyword-RRF |
| ------------- | ------------- |------------- |
| A.txt | 8 | 1/(_k_+8)=0.014925373 |
| B.txt | 10 | 1/(_k_+10)=0.014492754 |
| C.txt | None | 1/(_k_+?)=None |

**3, Σ RRF**
| FileName | calclated-RRF | displayed-RRF |
| ------------- | ------------- |------------- |
| A.txt | 1/(_k_+2)+1/(_k_+8)=0.031318816 | ≈0.03131881356239319 |
| B.txt | 1/(_k_+1)+1/(_k_+10)=0.031159420 | ≈0.031159421429038048 |
| C.txt | 1/(_k_+3)=0.016129032 | **≠0.029286926612257957** |

**Question 1.**
[The docs](https://github.com/Azure/cognitive-search-vector-pr/blob/main/docs/vector-search-overview.md#reciprocal-rank-fusion) describe _k_= like 60, but my calculation seems to be 59, how much is it actually?

**Question 2.**
"C.txt" was not found in the Simple Keyword Search (top=10) results. How is the RRF calculated in this case? My guess is that internally a larger value than I specified in my top query is specified and its ranking is calculated.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the linked Reciprocal Rank Fusion section in docs/vector-search-overview.md and compare its description with the reported keyword and vector rankings. Determine the actual constant and how documents missing from one result set are scored; done means the documentation answers both questions accurately, with the example reconciled.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure
Domain
documentation, search
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.