open-compass / open-compass/VLMEvalKit
AI2d gpt和claude3.5官方分数非常高
Open
Nobody has claimed this yet.
Awaiting Confirm
- Dominant language
- Python
- Stars
- 4.4k
- Forks
- 768
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 17
Description
您好～
想请问下你们对于openai和claude3.5在ai2d上特别高的分数有任何建议和想法吗？我这边修改姿势和prompt（添加cot）评测了gpt多次，都无法复现出0.942的超高分数。（加了cot后的最高分也就0.83），想请问你们对于这个gap有什么想法？（我看你们这边的ai2d的评测分数也没有任何高于0.9以上的，很好奇claude和gpt是怎么测出来将近满分的
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are identified. Start by comparing the AI2D evaluation setup and prompts used for OpenAI and Claude 3.5 with the reported 0.942 score; done means reproducing the result or explaining the gap.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, testing-qa
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100