Benchmark Gaming
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.5k
- Forks
- 4.8k
- PR merge metrics
- No merged PRs in 30d
Description
Hey there,
Thanks a lot for this super nice LLM benchmark!
Looking at the new release of Grok3 being clearly the top-1 model, I wondered whether you could make some analysis to figure out how much each company is biasing your benchmarks.
I heard from people in other companies that they use lmarena and can tell by the style of their model which model is theirs, and with grok this risk is even higher as it is more opinionated about politics. So, it should be even easier to tell for affiliated people.
A more regular release s.t. independent researchers can analyze the chats or analysis by you on this problem would be great to keep the benchmark the go-to.
Best,
Sam
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names no files, tests, or entry points. Start by locating the benchmark and Chatbot Arena analysis and release workflow, then define how model-affiliation bias would be measured and what independent data release is requested. Done would require an agreed analysis or release plan, but the current request remains broad.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100