API or data blob access to leaderboard output
- Dominant language
- Python
- Stars
- 39.5k
- Forks
- 4.8k
- PR merge metrics
- No merged PRs in 30d
Description
I asked on the Discord, and I believe that this isn't available today. If I am wrong, happy if you can direct me to the source.
At my company, I am responsible for compiling a list of (non-proprietary) models that our product is compatible with. I thought it would be nice if I could query the contents of the Leaderboard via an API, rather than picking through the list on a browser and copying and pasting.
Is the data that populates the current version of the leaderboard rankings available via an API, or an export of that data available as a flat-file in some manner?
There are examples of similar properties doing this, for example, MLperf releases their data ([example](https://public.tableau.com/app/profile/data.visualization6666/viz/MLCommons-InferenceDatacenter/MLCommons-Inference) )under Tableau, I'm hoping this isn't an unreasonable ask. I'm not asking for anything as fancy as Tableau, not even an API. A read-only CSV on AWS S3 or similar would be fantastic.
Right now, the site uses CloudFlare Captcha to prevent screen-scraping, which I respect. But it feels like making a limited set of data available would prevent people from either trying to circumvent captcha and/or copying and pasting a bunch of data by hand.
I _think_ what I am asking is that the leaderboard-table-file ([gradio_web_server_multi.py#L337](https://github.com/lm-sys/FastChat/blob/9a295b64ce3491ff15901f2d00f5e304b0ee78dc/fastchat/serve/gradio_web_server_multi.py#L337)) ([docs](https://github.com/lm-sys/FastChat/blob/9a295b64ce3491ff15901f2d00f5e304b0ee78dc/docs/commands/webserver.md?plain=1#L33)) is copied to an S3 bucket, or even just served raw via the site's http server (sans captcha, please), so that we can look at the raw data.
Contributor guide
No contributing guide indexed for this repository
Research direction
Read fastchat/serve/gradio_web_server_multi.py around the leaderboard-table-file reference and docs/commands/webserver.md around its configuration. Determine how the leaderboard data is currently produced and which read-only access format is appropriate. Done means the current leaderboard data is available through a documented API or export without requiring CAPTCHA.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100