open-compass / open-compass/VLMEvalKit
ChartQA 评测中存在误判问题
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.4k
- Forks
- 768
- Avg merge
- 2d 27m
- Merged PRs (30d)
- 18
Description
问题描述
在对 ChartQA 进行评测时，我发现评测框架在部分情况下会因为数值格式差异而出现误判。
示例
问题：
What's the percentage of U.S adults who refused?
图表：
模型回答：
2%
标准答案：
2
评测结果：
False
说明
该示例中，模型输出的 “2%” 与标准答案 “2” 实际上语义一致，均表示 2%。此类情况应视为正确答案。
如果这是我个人评测配置的问题，也请官方团队指正，非常感谢！
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the ChartQA evaluation and its answer-comparison or normalization logic. Reproduce the provided example where the model answer is “2%” and the standard answer is “2”; done means semantically equivalent percentage formats are evaluated as correct. No file or test path is named in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-visualization, machine-learning, testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100