Add Whole History Rating to Leaderboard?

For the API-based models, there are frequent claims online that users see models getting worse over time. It would be good to know if that's true. Copying a comment of mine from HF:

I know there are a bunch of Elo variants, but never learned the exact differences. Here is one summary:

To make a long story super short, Elo is the grandfather of most like systems. It has been around for ages and is super simple. Elo doesn’t care about inactivity or inconsistencies. The process starts from day 1 and moves chronologically throughout time, every competitor starts with a starter rating, which is then modified with each result. Glicko-1 is a very similar system to Elo, except it has the concept of “rating deviation” which allows competitors’ ratings to deviate more or less, given when they fought last. There is also a second version of Glicko, which tosses in a factor called volatility — it is a major complication with extremely limited benefit.

In comes WHR. Again, it is based on Elo, but is setup to take numerous passes throughout history. With each pass, it “learns” from what happened in surrounding events. This makes it an excellent system for reviewing the past and in trying to determine when a competitor was really at their peak. Whether it paints a more accurate ranking picture… who knows?

I know Glicko has a measure of uncertainty built-in, not sure how that compares to lmsys' bootstrap method.

Maybe WHR would be a better choice? I know WHR is used to track rock climber skill over time, for instance. From their own paper, they say:

Experiments demonstrate that, in comparison to Elo, Glicko, TrueSkill, and decayed-history algorithms, WHR produces better predictions.

WHR can show how models change in skill over time, and how confident we can be in the measurement:

The issue does not identify files, tests, or an implementation entry point. Start by locating the leaderboard rating calculation and its existing bootstrap or Elo-related code, then compare the proposed WHR approach with current outputs. Done means a decided rating method is implemented and the leaderboard can show historical skill and confidence for API-based models.