open-compass / open-compass/opencompass
[Feature] Support F1-score for DROP dataset
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.5k
- Forks
- 869
- Avg merge
- 17h 52m
- Merged PRs (30d)
- 13
Description
描述该功能
谷歌的Gemini, Claude 3 发布的页面上，DROP使用都是F1-score
是否希望自己实现该功能？
- 我希望自己来实现这一功能，并向 OpenCompass 贡献代码！
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the DROP dataset evaluation entry point in OpenCompass, then compare its current metric handling with the linked AllenNLP implementation at drop_em_and_f1.py. Confirm the expected F1 behavior on DROP examples and add or update the relevant evaluation coverage so the dataset reports F1-score.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100