How does calibrating for an anchor model in `elo_analysis` affect the elo ratings and bootstrap CIs?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.5k
- Forks
- 4.8k
- PR merge metrics
- No merged PRs in 30d
Description
Hi, I've been searching for an explanation for the choice of scaling the elo scores based on one of the models in `elo_analysis` when calculating the MLE elo ratings, but did not find any. Specifically, I see this line added at the end of `compute_elo_mle_with_tie`:
```
if "mixtral-8x7b-instruct-v0.1" in models.index:
elo_scores += 1114 - elo_scores[models["mixtral-8x7b-instruct-v0.1"]]
```
This effectively anchors the scores to mixtral-8x7b-instruct-v0.1 with a rating of 1114. Any explanation for the choice of the model and the seemingly arbitrary number 1114? How does adding this affect the overall elo ratings and corresponding bootstrap CIs? Is it chosen so that elo ratings can be compared over time? I'm asking because I noticed that the bootstrap CIs become a lot wider with the anchor model and I'm not sure which model to choose and number to set for my own set of models.
Any help will be appreciated, thanks!
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the `compute_elo_mle_with_tie` entry point in `elo_analysis`, including the anchor adjustment for `mixtral-8x7b-instruct-v0.1`. Trace how the bootstrap confidence intervals are calculated, then document why the model and 1114 rating are used, how the shift affects ratings and intervals, and how users should choose an anchor.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100