lm-sys / lm-sys/FastChat

How does calibrating for an anchor model in `elo_analysis` affect the elo ratings and bootstrap CIs?

Open
#3,377 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
39.5k
Forks
4.8k
PR merge metrics
No merged PRs in 30d

Description

Hi, I've been searching for an explanation for the choice of scaling the elo scores based on one of the models in `elo_analysis` when calculating the MLE elo ratings, but did not find any. Specifically, I see this line added at the end of `compute_elo_mle_with_tie`:
```
if "mixtral-8x7b-instruct-v0.1" in models.index:
elo_scores += 1114 - elo_scores[models["mixtral-8x7b-instruct-v0.1"]]
```

This effectively anchors the scores to mixtral-8x7b-instruct-v0.1 with a rating of 1114. Any explanation for the choice of the model and the seemingly arbitrary number 1114? How does adding this affect the overall elo ratings and corresponding bootstrap CIs? Is it chosen so that elo ratings can be compared over time? I'm asking because I noticed that the bootstrap CIs become a lot wider with the anchor model and I'm not sure which model to choose and number to set for my own set of models.

Any help will be appreciated, thanks!

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the `compute_elo_mle_with_tie` entry point in `elo_analysis`, including the anchor adjustment for `mixtral-8x7b-instruct-v0.1`. Trace how the bootstrap confidence intervals are calculated, then document why the model and 1114 rating are used, how the shift affects ratings and intervals, and how users should choose an anchor.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.