lm-sys / lm-sys/FastChat

API or data blob access to leaderboard output

Open
#3,730 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
39.5k
Forks
4.8k
PR merge metrics
No merged PRs in 30d

Description

I asked on the Discord, and I believe that this isn't available today. If I am wrong, happy if you can direct me to the source.

At my company, I am responsible for compiling a list of (non-proprietary) models that our product is compatible with. I thought it would be nice if I could query the contents of the Leaderboard via an API, rather than picking through the list on a browser and copying and pasting.

Is the data that populates the current version of the leaderboard rankings available via an API, or an export of that data available as a flat-file in some manner?

There are examples of similar properties doing this, for example, MLperf releases their data ([example](https://public.tableau.com/app/profile/data.visualization6666/viz/MLCommons-InferenceDatacenter/MLCommons-Inference) )under Tableau, I'm hoping this isn't an unreasonable ask. I'm not asking for anything as fancy as Tableau, not even an API. A read-only CSV on AWS S3 or similar would be fantastic.

Right now, the site uses CloudFlare Captcha to prevent screen-scraping, which I respect. But it feels like making a limited set of data available would prevent people from either trying to circumvent captcha and/or copying and pasting a bunch of data by hand.

I _think_ what I am asking is that the leaderboard-table-file ([gradio_web_server_multi.py#L337](https://github.com/lm-sys/FastChat/blob/9a295b64ce3491ff15901f2d00f5e304b0ee78dc/fastchat/serve/gradio_web_server_multi.py#L337)) ([docs](https://github.com/lm-sys/FastChat/blob/9a295b64ce3491ff15901f2d00f5e304b0ee78dc/docs/commands/webserver.md?plain=1#L33)) is copied to an S3 bucket, or even just served raw via the site's http server (sans captcha, please), so that we can look at the raw data.

Contributor guide

No contributing guide indexed for this repository

Research direction

Read fastchat/serve/gradio_web_server_multi.py around the leaderboard-table-file reference and docs/commands/webserver.md around its configuration. Determine how the leaderboard data is currently produced and which read-only access format is appropriate. Done means the current leaderboard data is available through a documented API or export without requiring CAPTCHA.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.