Replace Elo rating with a rating system of higher predictive power

Currently, the Chatbot Arena ranking appears to be based on a straightforward implementation of the classic Elo system used in chess and other two-player games.

This leaves a lot of value on the table. A better ranking could be extracted from the same data by changing the rating system.

There are two main issues with using Elo for Chatbot Arena:

First, the predictive power of the Elo rating system (that is, the ability to statistically predict the outcome of matches between opponents based on their ratings) has long been known to be sub-optimal. Popular alternatives such as Glicko-2 exist, and Kaggle once ran a competition to devise statistically superior rating systems (in which I participated and won a prize).

Second, and most importantly, Elo-like systems are inherently designed to rank players whose skill changes over time. That's why things like the K-factor exist, to allow novice players' ratings to quickly adapt as they rapidly improve at the beginning of their careers. Those assumptions don't make sense for language models, which don't change over time (else they'd be different models, by definition). With Elo, the final rating even depends on the order in which matches are tallied, which is obviously wrong for this use case.

A rating system specifically designed for the needs of Chatbot Arena could yield a more accurate ranking that better reflects the relative performance of models. If you are interested in such a radical change, I'd be happy to have a go at this. I know from experience that Elo can easily be beaten in terms of predictive power, and I would be able to back this up with numbers that demonstrate the statistical superiority of any proposed alternative.

Start with fastchat/serve/monitor/elo_analysis.py, the referenced Elo implementation, and review how Chatbot Arena match data is converted into rankings. Done would require choosing and implementing a replacement rating system, then demonstrating that it improves predictive power and does not depend on match-processing order.