open-compass / open-compass/VLMEvalKit
OmniDocBench 默认的是哪一个版本，实测数据与官方数据对不上
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.4k
- Forks
- 768
- Avg merge
- 2d 27m
- Merged PRs (30d)
- 18
Description
Hi，请问OmniDocBench默认测试的是一个版本，我使用Qwen2.5-VL-3B-Instruct，测得最终得分0.36，如下所示：
`
Edit_dist:
ALL_page_avg 0.360609
`
而Qwen2.5-VL技术报告中，OmniDocBench的en/zh版本测试结果为0.409/0.543，不管哪个版本和ALL_page_avg =0.36都不大能对上。
Looking forward to your reply, thank you！
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No file, test, or entry point is named in the issue. Start by locating the OmniDocBench evaluation path and its default dataset or version selection, then compare the reported ALL_page_avg value with the English and Chinese results in the Qwen2.5-VL technical report. Done means identifying the source of the discrepancy and documenting or correcting the relevant behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100